Compare commits

...
Author SHA1 Message Date
Daniel Salazar 3160b99f11 chore: bump ver (#4335) 2026-10-25 08:24:40 -07:00
Daniel Salazar 589a98b12c feat(kv): share KV prefixes with other users at watch/read/write, and bound what apps can be granted (#4331)
* feat(kv): share key-value prefixes with other users, and bound what apps can be granted

- KV share handles move out of Events into KvShareService, served at /kv/shares
  (the /events/kv-handles routes stay as an alias).
- Shares have watch, read and write levels, encoded in the permission root
  (kv-share, kv-share-read, kv-share-write) with write implying read and watch.
- Every puter.kv method takes optConfig.handle; keys are relative to the shared
  prefix, cursors are sealed, and private entries stay hidden.
- New puter.kv.share, unshare, listShared and listSharedByMe; optional label;
  kv.share.received notice in the app's mailbox only; recipient blocks apply.
- puter.kv.set gains onlyIfAbsent.
- fs and kv events gain viaWorker, and kv events report self from the writer.
- /auth/grant-user-app only accepts canonical permissions from a fixed set of
  families; the consent dialog, puter.perms and docs follow.
- Migrations: kv_share_handles gains revoked_by, label, and grantee and app
  indexes (mysql_mig_49, postgres_mig_38, sqlite 0095).

* fix(kv): never name a private key in a cross-app or handle cursor, and bound flat-permission scans

- Listings of another namespace (cross-app or through a handle) resume after
  the last entry they returned, and their cursors are sealed, bound to the
  namespace or handle, and padded to one length. Old plain cross-app cursors
  are still read when they name the target namespace; other namespaces get 400.
- PermissionStore scans flat permissions a few roots at a time and skips
  roots nested under another root in the same call.
2026-10-25 08:16:19 -07:00
Toshit ChawdaandReynaldi Chernando ec3ad1cb2f docs: easy/medium/hard networking recipes (#4040)
* add net category + http request recipes

* add tcp/tls socket recipes

* add complex recipes

* minor refine

---------

Co-authored-by: Reynaldi Chernando <[email protected]>
2026-10-25 19:22:45 +07:00
Reynaldi ChernandoandDaniel Salazar c7cc64e45c docs: medium events recipes (PUT-1971) (#4333)
* docs: easy events recipe (PUT-1961)

* docs: catch-up-on-notifications events recipe (PUT-1961)

* docs: events recipe corrections from a run against a live server (PUT-1961)

* docs: medium events recipes (PUT-1971)

* docs: simpler wording in medium events recipes (PUT-1971)

* refine

---------

Co-authored-by: Daniel Salazar <[email protected]>
2026-10-25 19:18:55 +07:00
Nariman Jelveh 0c67499dc2 fix(dashboard): serve stale profile pictures to every caller, skip detached roots
While a picture refreshed after its TTL, only the first caller got the
stale answer; every later one waited on the network, so in Files → Shared
one row showed the picture at once and the rest fell back to initials.
paintAvatars now also folds a frame's roots into one pass and skips roots
re-rendered away before the frame, so they fire no lookups.
2026-10-24 19:05:54 -07:00
Nariman Jelveh 34d74d36bd feat(dashboard): show profile pictures on people avatars (#4328)
* feat(dashboard): show profile pictures on people avatars

Avatars that name a person now show that person's profile picture once
it loads, and keep the initial-and-color avatar when there is none:

- Files → Shared: the "Shared by" cell and the people filter chips.
- Share dialog: the owner row and each person with access. Teams,
  invited emails and link shares keep their initial.
- Properties: a new Owner row with avatar for items someone else owns,
  and avatars beside each version's author.

Lookups go through one cache: each person is looked up once and shared
by every avatar of theirs, answers (including "no picture") are kept
for five minutes, at most 100 are kept, and at most four run at once.
Calls made while a listing renders row by row fold into one pass per
frame. The signed-in user's own picture is read from their loaded
profile, so it takes no request and follows changes in the Account tab.
The picture is set as an image's src property, never through markup.

* fix(dashboard): profile picture review — stale refresh, picker avatars, limiter

- Serve a stale picture while a fresh one loads, so a listing re-rendered
  after the cache expires no longer drops everyone back to initials.
- Show pictures on people in the share recipient suggestions, matching the
  rows of the dialog they open from.
- Free a lookup slot when a lookup throws before returning a promise.
- Move profilePictures.js to helpers/, now that a shared helper uses it.
2026-10-24 19:01:05 -07:00
Nariman Jelveh 0bdc2197b0 feat(dashboard): show who shared each item in Files → Shared (#4327)
* feat(dashboard): show who shared each item in the Files tab's Shared view

The Shared view now names the person behind every item:

- List view gets a sortable, resizable "Shared by" column with the
  share modal's initial-and-color avatar. The tooltip names the owner
  too when someone else re-shared the item.
- Grid tiles show the sharer under the item name.
- Narrow listings lead each row's second line with the sharer.
- A people filter above the listing (one chip per sharer, with counts)
  narrows the view to one person. It shows once two or more people have
  shared something, and scrolls sideways on small screens.

Sorting by "Shared by" applies only in the Shared view; other folders
keep sorting by name while it is the saved sort.

* fix(dashboard): keep the name column readable beside "Shared by"

In a listing between the phone layout and about 700px wide, the fixed
160px "Shared by" column squeezed an auto-width name down to a few
characters. Beside an auto-width name, "Shared by" now takes at most 40%
of the room the two split, and resizing it stops at the same width.

* fix(dashboard): keep the "Shared by" header on one line

Narrowed, its two-word label wrapped onto a second line that spilled out
of the 32px header row. It now stays on one line and ends in an
ellipsis, as longer translations need too.

* fix(dashboard): fit "Shared by" to its longest name on double-click

Auto-fitting the column measured names in a font the cell doesn't use
and left out part of its padding, so the longest name still ended in an
ellipsis. It now reads the font, padding, avatar and gap off a rendered
cell.

* fix(dashboard): keep the people filter on coming back to Shared

Opening a folder from a filtered Shared view and going Back reset the
filter to Everyone, so going through one person's folders meant picking
them again each time. The filter now lasts while the user browses what
was shared with them, and clears once they go back to their own files.

* fix(dashboard): update Shared after Remove from Shared

Remove from Shared only took the row away, leaving the people filter
counting the item. Removing the last item of the person filtered to left
a blank pane under a chip that still claimed one. Shared is now listed
again afterwards, which updates the filter, the empty notice and the
footer.

* fix(dashboard): show a notified share past the people filter

Opening a "shared with you" notification, or a share link, while Shared
was filtered to someone else picked out nothing: the item was filtered
away. The filter now gives way to everyone's when it hides an item the
notification or link names.

* fix(dashboard): truncate a long sharer name on phone-width rows

In the two-line row layout, a long username grew the details line past
the row, pushing the date off screen and under the ⋯ button. The name
now ends in an ellipsis, and the line keeps clear of the button as the
name line does.

* perf(dashboard): sort the Shared view by sharer with one collator

Each sharer comparison called localeCompare with options, which builds
a collator per call: sorting 2,000 rows took about 25ms in Chrome. One
shared Intl.Collator orders them the same in under 1ms.

* fix(dashboard): make share avatar initials readable for every name

The initial-and-color avatar drew its letter at too little contrast for
nearly half the hues a username can map to: yellows and greens came out
near 3:1 in light mode, and lower still on a selected row. The letter is
now darker in light mode and lighter in dark mode, reaching 4.5:1 for
every hue on the page, selected rows and the share suggestions' hover
row. The share dialog and the Shared listing change together, so a
person keeps one look.

* fix(dashboard): show every folder when a drag springs into Shared

Dragging an item out of a shared folder and holding it over Up, Back or
the Shared crumb opens Shared to drop into a folder there. The people
filter, which now lasts while browsing shared items, could hide those
folders, and its chips can't be pressed mid-drag. A drag arriving in
Shared now sees everyone's folders.

* fix(dashboard): keep filtering Shared when it can't be listed again

A chip click lists Shared again so its rows are current. When that
request failed, the listing was replaced with "This folder couldn't be
opened" and the chips disappeared, though only a filter had changed. A
filter change now applies to the rows already loaded when the listing
can't be fetched.

* fix(dashboard): return a drag let go over Shared to where it began

A drag that springs into Shared and is let go over the listing rather
than a folder tried to move the items into Shared itself, which is a
query and not a folder: nothing moved, nothing said so, and the user was
left in Shared. Let go over a listing that takes no items, the drag now
goes back to the folder it started in, as the cancel zone does.

* fix(dashboard): don't offer read-only shared folders as drop targets

Folders shared read-only lit up under a drag and took the drop, which
the server then refused with "Forbidden". They no longer light up, and
a drop on one sends the drag back where it began; they still open on a
dwell.

* fix(dashboard): slide a drag back when a read-only folder refuses it

A drop a read-only shared folder refused left the drag chip to vanish
where it was let go, as if something had happened. It now slides back
to where the drag began, as any drop that doesn't land does.
2026-10-24 18:10:27 -07:00
Nariman Jelveh 7759f7afe9 fix(dashboard): seventh adversarial review — bugs, UX, a11y, perf (#4326)
* fix(dashboard): show notification times in Brazilian Portuguese, Ukrainian and Traditional Chinese

The notification panel and toasts handed Puter's language code straight to
Intl, and a few of those codes aren't BCP 47 tags. Brazilian Portuguese
(`br`) is Breton to Intl, so times read "5 munut zo"; Ukrainian (`ua`)
fell back to the browser's default language; and `zhtw` threw, so Taiwan
users got English. Those codes are now mapped to pt-BR, uk and zh-TW
before formatting.

* fix(dashboard): count "yesterday" and "N days ago" in calendar days

Notification times rounded the elapsed hours to whole days, so at 23:30
something from 00:45 the same day read "yesterday", and on Wednesday
morning something from Monday night (34 hours earlier) read "yesterday"
too. Days are now counted from midnight to midnight in the reader's time
zone; anything from today stays in hours.

* fix(dashboard): don't let deploy results crowd notifications out of the panel

The panel listed the 30 newest notification rows and then dropped the
ones it never shows, a worker's deploy results among them. A developer
with 30 deploys since their last share saw "No notifications" and no
badge while the share sat behind them. The listing now asks for up to
200 rows and keeps the newest 30 the panel shows.

* fix(dashboard): keep every unread notification in the panel

The socket replays up to 200 unread notifications on connect, but the
panel's listing held only the 30 newest, and the first refresh after that
dropped the rest: 45 unread read "45" until the panel was opened, then
"30", and "Mark all as read" left the other 15 unread on the server for
the next visit. The panel now keeps unread notifications past the newest
30, up to the 200 a listing returns.

* fix(dashboard): let Undo take back a new Files folder or file

The Files tab creates folders and New › files itself rather than through
the desktop's helpers, and never recorded them for Undo. Undo then
reversed whatever came before: upload notes.txt, create a folder, press
Ctrl+Z, and notes.txt was deleted for good (undoing an upload deletes it
outright, not to Trash) while the new folder stayed. Both now record
themselves, so Undo removes what was just created, as on the desktop.

* fix(dashboard): let Undo take back a Files cut and paste

Pasting cut items in the Files tab moves them itself rather than through
the desktop's move_items, and recorded nothing for Undo. Undo after a
paste reversed the action before it instead, which deletes an upload for
good. The paste now records the move as move_items does, so Undo puts
the items back where they were cut from.

* fix(dashboard): don't drop a Files selection onto one of its own folders

With folders A and B selected, dragging A onto B moved A into B and then
stopped on "Cannot move a directory into itself" for B. Hovering B also
sprang it open mid-drag. A folder row that is part of the drag is no
longer a drop target, so the drag reverts as it does over any other
non-target. A drop elsewhere (a sidebar folder, say) also leaves out the
target folder itself and any folder that holds it.

* fix(dashboard): don't drop a Files refresh asked for during another

A refresh of the folder on screen was ignored while another refresh of
it was loading. Drop two files: when the second finished during the
first's refresh, its own refresh was dropped, the earlier listing (read
before the file existed) was drawn, and the file's item.added was
ignored meanwhile too, so it stayed missing until a manual refresh.
Paste, Refresh and returning to the browser tab could lose items the
same way. A refresh now takes over from one in flight, as opening
another folder already did; a navigation in progress still holds.

* fix(dashboard): keep the phone selection bar after cancelling a delete

In select mode on a phone, tapping Delete permanently and then Cancel
left every item selected but hid the floating action bar, the only way
to act on the selection, until the selection next changed. The same
happened when nothing in the selection could be moved to Trash. The bar
now goes away only once something is actually deleted or trashed.

* fix(dashboard): show the phone selection bar when Select starts with a selection

Uploaded or pasted items arrive selected. Tapping Select on a phone then
turned select mode on with them selected but no floating action bar,
until a row was toggled. Entering select mode now brings the bar up for
the selection it starts with.

* fix(dashboard): give Files footer sizes in the rows' format

The footer used the desktop's byte formatter while the rows use their
own, so a 5-byte file read "5 B" in its row and "5.00 Bytes" below it,
1.5 KB read "1.50 KB", and a folder of folders read "0 Byte". The footer
now formats sizes as the rows do.

* fix(dashboard): title a trashed folder's Properties with its name

Properties from the breadcrumb of a folder opened inside Trash was
titled with the folder's uid, the last segment of its trashed path,
while the crumb itself showed the original name. It now uses that name.

* fix(dashboard): show delete and log-out confirmations without entities

The Files tab's "delete these items permanently?" confirmation and the
dashboard's "close your apps and log out?" one passed HTML-encoded i18n
strings to UIAlert, which encodes its message and button labels again.
In Igbo both questions read "Ị ji n&#039;aka …". These alerts, and the
Files tab's name-conflict and folder-depth ones, now pass plain strings.

* fix(gui): read the Trash's state fresh when updating its icon

refresh_trash_state stats the Trash to pick the empty or full icon, and
its comment calls for a strong read, but it asked for an eventual one.
Once a listing of Home had cached the Trash entry (without `is_empty`),
the stat came back without it: permanently deleting the last items from
Trash in Files left the full-Trash icon up, and other tabs were told the
same. The stat is now strong, as intended.

* fix(dashboard): don't add a Files row to a folder opened meanwhile

A file added from elsewhere (an upload, another tab, an app) is checked
against the folder on screen and then drawn once its icon resolves. A web
link's or app's icon is read from the file, and opening another folder
during that read put the row into the new folder's listing, where it
stayed and opened the file from the old one. The checks now run again
once the icon is in.

* fix(dashboard): let a focused folder or crumb take the Menu key

The Files tab answered the Menu key and Shift+F10 with the selection's
menu wherever focus was. With a file selected and a sidebar folder, a
breadcrumb or Home's Shared row focused, the key opened the file's menu;
with nothing selected it cancelled the browser's contextmenu, so the
focused element opened none. A focused control now gets the browser's
contextmenu event, and its own menu, as with a right-click.

* fix(dashboard): don't drop New Folder during a Files refresh

New Folder did nothing while the folder on screen was refreshing (after
an upload or paste, or on returning to the browser tab). A refresh keeps
the rows up with no spinner, so the click just looked ignored. It now
takes over the refresh and makes the folder once the listing is in;
during a navigation to another folder it is still dropped, as before.

* fix(dashboard): step through Files names on a repeated letter

Type-to-select meant to move past the selected row to the next name with
the same start, but returned early whenever the selected row matched, so
the stepping never ran: with readme, report-a and report-b, pressing "r"
again stayed on readme, and "rr" matched nothing. A repeated letter now
steps through the names it starts, wrapping around, as file managers do;
a longer prefix still stays on a selected row it matches.

* fix(dashboard): open one touch menu per long-press

A long-press on Android fires both contextmenu and taphold. The Files
rows already ignore the second, but a breadcrumb, a sidebar folder,
Home's Shared and Trash rows and the listing's empty space each opened a
second sheet over the first; picking an item or tapping the backdrop
then left the other one up. The touch menu now doesn't open while
another is on screen.

* fix(dashboard): treat Ctrl+click on a Mac as a right-click in Files

On a Mac, Ctrl+click opens the context menu, but the Files rows read it
as a Ctrl-toggle first: Ctrl+clicking one of three selected files
deselected it and the menu then acted on that file alone, and on an
unselected file it was added to the selection. On a Mac it now leaves
the selection alone, as a right-click does; Cmd+click still toggles.

* fix(dashboard): keep arrow keys in the open phone drawer on Apps

With the drawer open over the Apps tab, an arrow key moved focus out of
it to the first app tile behind the scrim, and the next Tab then walked
the page behind it. The open drawer now keeps the arrow keys, as the
notifications panel and dialogs already do.

* fix(dashboard): act once on a held Enter or Space in Apps

Holding Enter on an Apps folder opened it, moved focus to its first app,
and the key's repeats then launched that app; held on the "+" tile, the
repeats picked "Browse the App Center" in the dialog it opened, and held
in the search box they launched the first match again and again. Key
repeats now do nothing on a tile, in the search box or on the add-app
dialog's options.

* fix(dashboard): don't blank the Apps grid when it re-flows on resize

When a resize changed how many columns or rows fit (resizing the window,
collapsing the sidebar, zooming, an Android keyboard), the grid was
rebuilt behind the first load's fade, which waits on page one's icons.
With an icon host that was slow or unreachable the grid went blank for
up to 1.5s each time. A grid already on screen now re-flows at once;
one still loading keeps its fade.

* fix(dashboard): open an app from the add-app dialog at once during a search

"Browse the App Center" and "Build with AI" play an intro from the
app's tile, and wait up to 3s for that tile to appear. With a search
that leaves the app out (the case the "+" tile stays for: a search that
found nothing), the tile never appears, so the app opened only after the
full 3s. The intro now gives up as soon as the search is what hides the
tile, and the app opens straight away.

* fix(dashboard): make the Apps search's clear control a button

The "×" that clears the Apps search was a bare SVG with a click handler:
Tab couldn't reach it, screen readers had nothing to announce, and on a
phone it was a 16px target inside the field. It is now a labelled button
after the field in the Tab order, with a focus ring, and on touch it
hit-tests at 44x44 while keeping its look.

* fix(dashboard): tell screen readers when an Apps search finds nothing

Typing in the Apps search re-renders the grid silently, and the "No apps
match your search" line is drawn into it fresh each time, which screen
readers don't announce. A search that matched nothing gave no feedback
at all. A persistent status region now says so.

* fix(dashboard): keep keyboard focus on an Apps tile across a re-render

The grid is rebuilt wholesale whenever its data or size changes: the
refresh that follows opening the tab, a resize or zoom that changes how
many columns fit, an uninstall that fails. A keyboard user who had
arrowed onto a tile lost focus to <body>, so Enter did nothing and the
next arrow started over from the first tile. Focus now moves to the
same app's or folder's new tile.

* fix(dashboard): don't offer Change username to a temporary account

A temporary account's Account tab offered Change username, whose dialog
asks for an account password the account doesn't have: left empty it
said "All fields are required", and anything typed got a 403 from the
server, which refuses temporary accounts. The button is now left out
until the account is saved, as the Password card already is; saving it
re-renders the tab with the button.

* fix(dashboard): don't offer Change email to a team seat

A team seat's Account tab offered Change email, and after the user
filled in the new address and their password the server refused it:
"Your team set this address. Ask a team admin to change it." The Email
card now shows that note in place of the button, as the Username card
already does for seats.

* fix(dashboard): don't let a slow picture removal undo a newer picture

Saving a profile picture is ordered so an older save can't put its
picture back over a newer one, but Remove picture stood outside that
order. Clicking Remove on a slow connection and then saving a new photo
before the removal answered left the default avatar on screen, and
Remove hidden, while the account had the new photo. A removal now
yields to a picture saved after it.

* fix(dashboard): show placeholders in the Usage headers until they load

The Storage and Resources headers started out as " ·  used of " with the
numbers left blank, and stayed that way when reading the storage space
or the month's usage failed. They now start at "--% · -- used of --",
as Home's cards start at "--", and keep that if the read fails.

* fix(dashboard): write Home's trial dates in the dashboard's language

The free-trial note on Home formats its dates with the browser's
language, not the one chosen in Puter, so with Puter in French and the
browser in English it read "…le Mar 5", and the other way round "First
charge on 5 mars". The dates now follow the dashboard's language. The
notification panel's mapping of Puter's language codes to Intl tags
(Brazilian Portuguese, Ukrainian, Traditional Chinese) moves to
i18n/intlLocale.js so both use it.

* fix(dashboard): write credit and dollar amounts in the dashboard's language

Credits were always formatted with English separators and dollar amounts
with "$" and English separators, so in German Home read "5,000
monatliche Credits", which a German reader takes as five, and Usage
showed "2.5" and "$1,234.50". Both now use the dashboard's language
("5.000", "2,5", "1.234,50 $"); English is unchanged.

* fix(dashboard): say why Create token is off where everyone can read it

When the account's email isn't confirmed, Create token is disabled and
the reason ("Verify your email address to create an API token.") lived
only in a hover tooltip: a disabled button isn't reachable by Tab, a
phone has no hover, and screen readers don't read a plain span's title,
so those users met a dead button with no explanation. The reason now
replaces the card's description while the button is off. The tooltip,
which encoded the already-encoded text again, now uses it as is.

* fix(dashboard): announce that a new API token is shown only once

After a token was created, focus moved to Done and a screen reader read
"Done, button" and nothing else: the done state arrives already filled
in a status region, which isn't announced, and Done is described by the
message only for plain-text results. The token dialog's warning ("Copy
this token now — it won't be shown again…") now describes Done, so it
is read out when focus lands there.

* fix(dashboard): show an email change confirmed in another tab

Confirming a change of email from its link sets the new address and
marks it confirmed, but sends no socket event, unlike confirming a code.
Back in the dashboard the user was re-read on returning to the browser
tab, yet nothing redrew: Account kept the old address under the
username, Create token stayed disabled and Security had no 2FA card
until a reload. Account and Security now redraw after that re-read when
the address or its confirmation changed.

* fix(dashboard): close the share dialog after leaving a share

Someone the owner let share could remove their own row in the share
dialog to leave. The removal went through, but the dialog then reloaded
its access list with the access just given up: the server refused it,
the success line gave way to "Subject does not exist" over an empty
list, and the item stayed in the Shared view until a reload. Leaving
now closes the dialog and refreshes the folder on screen, which drops
the item, as "Remove from Shared" does.

* fix(dashboard): say the access list couldn't load, not that sharing failed

When the share dialog couldn't read who has access (a network error,
which carries no message of its own), it said "Could not share this
item." though nothing had been shared or attempted. It now says "Could
not load who has access."

* fix(dashboard): don't pull focus back after a share change saves

When a change in the share dialog finished saving (a level change, Add
to all, a removal, a share), focus was put back on that row's control or
the add field, wherever the user had gone meanwhile. Someone who changed
a person's level and then started typing a name had the next keystrokes
land on that person's level select, where a matching letter picked
another level and granted it. Focus now goes back only when the save
took it away; focus the user moved elsewhere stays there.

* refactor(gui): move the localized time formatting to i18n/formatTime.js

formatRelativeTime and formatAbsoluteTime lived in the notification
center's module; Files and Properties are about to use them too.

* fix(dashboard): write Files and Properties dates in the dashboard's language

The Files tab's Modified column and Properties' Modified, Created and
version times came from timeago, which has no language but English
registered, so a German dashboard read "3 hours ago" beside "Geändert".
They now use the same localized formatter as the notification panel
("vor 3 Stunden"), which also gives a date past a week rather than
"3 weeks ago". Its formatters are now made once per language, since a
listing formats one date per row.

* fix(dashboard): style the Usage table's Show less button in dark mode

The button's dark variant was written for an `html.dark-mode` class that
nothing sets, so in dark mode it kept its light look: grey text at 3.4:1
on the dark card, a light border, and a light chip on hover. Its rules
move under prefers-color-scheme, as the table's fade overlay already
did.

* fix(dashboard): show the Files breadcrumb separators in dark mode

The chevrons between breadcrumbs are a black stroke at 20% opacity,
which vanishes on the dark header: a nested folder read "Home Documents
Work" with nothing between the names. In dark mode they are now
inverted, like the toolbar's icons.

* fix(dashboard): show the website-link badge on dark Files tiles

In grid view a file inside a published folder carries a link badge: a
black glyph on a disc of the page's background color. In dark mode that
is black on near-black and can't be seen. The badge now gets the white
disc the shortcut and worker badges already have.

* fix(dashboard): skip disabled Back, Forward and Up when tabbing in Files

The Files header's arrows stayed in the Tab order while disabled, drawn
at 10% opacity, which takes their focus ring with them. Tabbing into
Files, the first two stops (Back and Forward, disabled on arrival)
looked like nothing had focus. A disabled arrow now leaves the Tab
order, as a disabled button does, and returns once it can be used.

* fix(dashboard): give the Delete account button an AA-contrast red

The Account tab's Delete account button used the desktop's danger red
(#f00808), which puts its white label at 4.4:1, under the 4.5:1 that
WCAG AA asks of text. It now uses the dashboard dialogs' danger red
(#d90707, 5.3:1) and their darker hover, so the two match.

* fix(dashboard): label collapsed sidebar items for keyboard users too

With the sidebar collapsed, each item shows only its icon, and its name
appeared in a tooltip on mouse hover alone. Tabbing through the sidebar
showed a row of icons with no names. The tooltip now also shows while
an item has keyboard focus.

* fix(dashboard): keep Share and Properties inside the visible screen

Both dialogs capped their height at 85vh (88vh as phone sheets), and vh
is the viewport with the browser's bars hidden. With the bars showing,
on a phone in landscape say, the card plus the overlay's padding was
taller than the screen, and the overlay centers without scrolling: the
title and the buttons were cut off with no way to reach them. Each card
is now also held within its overlay, as the dashboard's dialogs are,
and its body scrolls instead.

* fix(dashboard): outline the off state of the 2FA switch

Off, the switch was a light grey track (#e0e0e0) with a white thumb,
1.3:1 against a white card and 1.2:1 against the amber card the 2FA
toggle sits on while 2FA is off, so the control could barely be seen
where it matters most. WCAG asks 3:1 for a control's boundary. The off
track now carries a 1px slate outline (4.3:1 on the amber card); the
on state is unchanged.

* fix(dashboard): make the keyboard focus ring visible on sidebar, tiles and lists

The sidebar items, the Files folders, crumbs and arrows, the app tiles,
Home's cards and recent apps, the notification rows and the Usage sort
buttons show keyboard focus with a 2px outline and nothing else, drawn
in --select-ring: the selection color at 40% (50% dark), about 1.6:1
against the surfaces it sits on. Keyboard users could hardly tell where
focus was; a focus indicator needs 3:1. Those outlines now use the solid
selection color (3.7:1 on white), as the switch and pager dots already
do. Field halos, which come with a colored border, are unchanged.

* fix(dashboard): bring text in the selection blue up to AA contrast

The selection blue is 3.7:1 against white, and a few places set text in
it or white text on it, under the 4.5:1 that text needs: the active
folder in the Files sidebar, the share dialog's "Add to all" link, the
add-app dialog's Send button, the notification count badge, the drag
count badge, and a breadcrumb whose menu is open. They now use the same
hue darkened to 45% (5.3:1; a lighter one for text in dark mode), from
two new tokens, --select-text and --select-fill.

* fix(dashboard): give the Apps page dots a 24px touch target

The Apps pager's dots, and a folder's, are 17px buttons side by side,
under the 24px a target needs when it has neighbors, so on a phone or
tablet a tap meant for one page often opened the next. On a touch
screen each dot is now 24px around the same 7px dot; mouse layouts are
unchanged.

* fix(dashboard): return from a drag-opened folder while it is still loading

Hovering a dragged file over a folder springs it open. Cancelling the
drag (Escape, the Cancel zone, or a drop outside the listing) is meant
to go back where the drag started, but while the sprung folder was still
loading the return was dropped like any navigation during another, and
the user was left in the folder they had backed out of. The return now
takes over from that load.

* fix(dashboard): drop into a drag-opened folder while it is still loading

When a dragged file springs a folder open, the drop anywhere in the
listing moves it into the folder now shown. Until that folder's listing
arrived, though, the Files tab still counted the folder the drag left as
the one on screen, so a drop in that time was taken as a move to where
the file already was and did nothing. It now goes to the folder being
opened.

* fix(dashboard): say "Incorrect password" when the account password is wrong

The account dialogs showed the server's message as is. A wrong current
password in Change password read "Password mismatch", in English in
every language, beside a dialog that says "New Password and Confirm New
Password do not match" when those two differ, so users corrected the
wrong fields. A wrong code read "Incorrect code." and too many attempts
"Too many requests.", also always in English. These now show the
translated messages the GUI already has, and a wrong password puts the
status on the password field, focused for retyping.

* fix(dashboard): don't send an account change to the value it already has

Change username with the account's own name got "This username is
already taken." and used up one of the hour's password attempts. Change
email with the current address was accepted: it mailed a confirmation
and a change notice to the same inbox, and following the link unlinked
any sign-in provider. Change password to the current password signed
out every other session and mailed a "password changed" notice. Each
dialog now says the value is already the account's and sends nothing.

* fix(dashboard): ask for the new password after a provider-only account sets one

An account that signs in only through a linked provider can set a
password with Change password, after which the server treats it as a
password account. This session's copy still said provider-only, so
Change username, Change email and Delete account went on hiding the
password field and sending the user through the provider's popup until
a reload. The copy now changes with the password.

* fix(dashboard): let Escape cancel a session rename without closing Manage sessions

In Manage sessions (Security), Escape in a session's rename field
cancelled the rename and then closed the whole modal, and Escape in the
search box closed it too, search and all. Escape now cancels the rename
or empties the search, and only an Escape with neither closes the
modal.

* fix(dashboard): show the server's message when a session can't be revoked

Manage sessions put a refused request's raw response body in its alert,
so revoking a session already signed out from another tab read
{"message":"Session not found","code":"not_found"}, and the stale row
stayed listed; a failed rename and "Revoke all other sessions" did the
same. The alerts now show the message itself, and a session found gone
reloads the list.

* fix(dashboard): show 2FA as it is when its dialog closes without success

The Security tab's 2FA toggle assumed nothing had changed whenever its
dialog closed without success. With 2FA turned on in another tab, the
setup here refused ("2FA is already enabled."), the toggle went back to
off, and every retry got the same refusal, with Disable never offered;
a code accepted just as the user closed the setup dialog left the same
state. The tab now re-reads the account and shows its real 2FA state.

* fix(gui): leave Ctrl/Cmd+Z to the text field being typed in

Ctrl/Cmd+Z ran the file Undo wherever focus was, and cancelled the
field's own text undo. Typing a new name (F2) or anything in another
field and pressing it undid the last file action instead; when that was
an upload, the uploaded file was deleted for good. Undo now leaves text
fields alone.

* fix(gui): let Escape on a context menu close only the menu

Escape removed any open context menu and then went on to the Files tab's
own Escape handling, which, finding no menu left, cleared the selection
and closed the image preview. Select three files, open their menu,
press Escape: the menu closed and the selection was gone. When Escape
closes a menu, nothing else now acts on it; a second Escape clears the
selection as before.

* fix(dashboard): close the Files image preview when an app opens over it

The image preview sits above every window, and only a double-click
closed it on the way to an app. Opening the image with Enter, or with
Open or Open With from its menu, started the app underneath the
preview, which went on covering it while keyboard focus moved into the
app, so Space and Escape no longer reached the preview. It now closes
whenever an app window comes on screen.

* fix(dashboard): close the Files image preview when its file goes away

Deleting the previewed image (the Delete key, the menu, the phone
selection bar), moving it, or having it removed from another tab took
its row out of the listing but left the preview up, still showing the
image. The preview now closes with its row; a refresh of the folder,
which redraws the row, keeps it.

* fix(dashboard): download one selected file as itself from the phone selection bar

In select mode, the floating bar's Download zipped whatever was
selected, so a single photo arrived as "photo.jpg.zip" where its own
menu's Download gives "photo.jpg". One selected file now downloads as
itself; folders and several items are still zipped.

* fix(dashboard): don't offer New Folder, Upload or Paste in a view-only share

Inside a folder shared with the user to view only, the Files tab still
showed New Folder and Upload, offered New and Paste in its menu, and
took files dropped on the listing or on a folder in it; each then
failed: an upload window ending in an error, or a new folder row with
its name field open, then taken back with a Forbidden alert. Those are
now left out there, as in Trash. A share whose level isn't known yet
stays as before, the server deciding.

* fix(dashboard): move through Files with Home, End, Page Up and Page Down

The Files listing took the arrow keys but not Home, End, Page Up or Page
Down, and as the listing itself never has focus the browser didn't
scroll it on them either: in a folder of hundreds of items the keyboard
could only step through them one row at a time. Home and End now select
the first and last item, Page Up and Page Down move a screenful, and
Shift extends the selection as with the arrows. With a menu open they
are left to it, as the arrows are.

* fix(dashboard): apply the hidden-files setting to a renamed Files row

With Show hidden off, renaming notes.txt to .notes.txt left the row in
the listing until the folder was next loaded, and with it on the row
wasn't dimmed as other dot-files are; renaming .env to env left it
dimmed. A rename made in another tab did the same. A renamed row now
follows the setting as listing the folder would.

* fix(dashboard): open a recent app once on a double-click

Double-clicking an app under Home's "Recently used" opened it in two new
browser tabs, one per click. The second click of a double-click is now
ignored, as on the Apps tab's tiles.

* fix(dashboard): don't say "No recent apps yet" before Home's list has loaded

Home's "Recently used" card was drawn from the list as it stood when the
page opened, which is empty until the request answers: every visit
first read "No recent apps yet / Apps you use will appear here" and
then swapped in the apps, and if the request failed the card kept
saying there were none. It now stays blank until the list arrives, and
a first load that fails says the apps couldn't be loaded; a list
already shown stays when a later refresh fails.

* fix(dashboard): drop "Save session" when the account was saved in another tab

A temporary account saved in one browser tab stayed temporary-looking
in its other dashboard tabs: Home kept its "Save session" button, which
opened Save Account for an account already saved, and Account and
Security kept their temporary-account layout. Coming back to such a tab
re-reads the account; that now removes the button and redraws the two
tabs when the account is no longer temporary.

* fix(dashboard): let go of the phone drawer when the window widens

With the sidebar open as a drawer below 768px, widening the window past
it (an iPad turned to landscape) drew the sidebar as the desktop one but
left it marked open: Tab stayed trapped in it, so the content couldn't
be reached by keyboard, and the Apps grid ignored the arrow keys, until
an Escape or a click in the content. The drawer now closes when the
window widens, leaving focus where it is.

* fix(dashboard): don't let a #files/<path> link pull Files back to its folder

Files doesn't keep the folder on screen in the URL, but a link like
/dashboard#files/alice/Documents stayed in the address bar once opened.
After the user moved on to Pictures, went to Home and pressed Back, the
browser returned to that link and Files jumped back to Documents; a
reload did the same, and Files' own Back/Forward lost track, as the jump
wasn't recorded in it. The link's path is now consumed once opened, as
a share link's is, and a folder opened from such a link is a step in
Files' history.

* fix(dashboard): close the drawer and Files' dialogs on browser Back and Forward

Browser Back or Forward switched tabs but left whatever was open on
screen: on a phone, Android's Back changed the tab under the open menu
drawer and left it open; in Files, the Share or Properties dialog
stayed up over Home, still acting on Files rows; an open context menu
stayed too. These now close with the tab, as with a sidebar click.

* fix(dashboard): fill Home's storage and usage meters without waiting on the plan

Home read the plan first and only then asked for storage and the month's
usage, so every visit paid for one more round-trip before either meter
filled in. The plan request had no timeout either: one that never
answered kept both meters at "--" and, as later loads wait for the one
in flight, held every later refresh of the cards for the rest of the
session. The three now load side by side, and the plan read gives up
after 15 seconds.

* fix(dashboard): link Home's card headers to their tabs

The headers of Home's cards (Recently used, Usage, Your Plan, Your
Storage, Resources) linked to "#", so opening one in a new browser tab
(middle-click, the browser's menu) showed Apps rather than the tab it
names, and Cmd/Ctrl+click switched tabs in place. They now link to the
tab they open, and a modified click is left to the browser, as on the
sidebar.

* fix(dashboard): prefill the username when saving a session from Home

Home's "Save session" button opened Save Account with an empty username,
so a temporary user had to make one up, while the same action in the
user menu fills in the account's current name and starts at the email
field. Both now open it the same way.

* fix(dashboard): measure Home's storage meter by the amount it shows

On a self-hosted install the storage card names the disk's use (Puter's
files and everything else on it) but filled its meter, and gave its
percentage, from Puter's files alone, so it could read "93 GB used ·
0% of 466 GB". The meter and percentage now come from the same amount
the card shows.

* fix(dashboard): refresh Home's recently used apps on returning to the tab

Recent apps open in a new browser tab, and coming back to the dashboard
refreshed the usage cards but not "Recently used", whose comment said
it followed the same triggers: the app just opened only moved up the
list after switching away from Home and back. Returning to the tab now
re-reads the list too.

* fix(dashboard): leave IME keystrokes to the IME in Apps

Typing Japanese or Chinese in the Apps search or a folder's name, the
keys that drive the input method reached the tab's own handlers:
ArrowDown to pick a candidate moved focus to the first tile and cut the
conversion short, Enter to confirm one named the folder half-written or
launched the first match, and Escape put the old name back. The search
box's Enter meant to skip composition, but read isComposing off jQuery's
event, which never carries it. Keys during composition are now left to
the IME.

* fix(dashboard): leave IME keystrokes to the IME in the share recipient field

With a Japanese or Chinese input method on, even a Latin username is
composed, and the share dialog's recipient field took the keys that
drive it: the arrows moved through its suggestions instead of the IME's
candidates, and Enter to confirm the text picked a suggestion instead.
Keys during composition are now left to the IME.

* fix(dashboard): re-read the Apps layout on returning to its browser tab

The Apps tab read its order and folders only when it was opened, and
saves write its whole copy of them. With the dashboard open in two
browser tabs, a folder made in one was wiped as soon as the other,
showing Apps since before, changed anything (a reorder, a new folder, a
rename), as was the order arranged in the first. The tab now reads the
layout again when its browser tab comes back into view.

* fix(dashboard): undo a drag out of an Apps folder when the drag is cancelled

Dragging an app past the edge of an open folder takes it out, closes the
folder and carries the drag on over the grid, and the removal was saved
right then. Pressing Escape, or the window losing focus, cancels the
drag but left the app out; out of a two-app folder that dissolved the
folder for good, its name with it (dragging the app back made a new
"Folder"). A cancelled drag now puts the folder and the order back.

* fix(gui): show the 2FA setup's confirmations without entities

The 2FA setup window, which the dashboard's Security tab opens, encoded
already-encoded translations for its labels, so in French the two
checkboxes before enabling read "J&apos;ai enregistré mes codes…" and
"Je suis prêt à activer l&apos;A2F". The labels, and the window's other
translated text set the same way, are now encoded once.

* fix(gui): don't leave the 2FA dialogs stuck when a request can't be sent

Turning 2FA on or off from the dashboard's Security tab while offline,
or on a dropped connection, left the dialog busy for good: enabling kept
its spinner up with the code boxes disabled, and disabling kept its
button and password field disabled, with nothing said either way. A
request that gets no answer is now reported like a refused one, and the
dialog can be tried again.

* fix(gui): stop disabling 2FA when the provider sign-in doesn't happen

For an account that signs in through a linked provider, turning 2FA off
opens the provider's sign-in popup. When the popup was closed or
blocked, the dialog showed why for a moment and then sent the request
anyway, whose refusal ("OIDC revalidation required") replaced the
reason. It now stops at the popup's error.
2026-10-24 15:05:56 -07:00
Filip KujundžićandClaude Fable 5.1 bbb780eb31 feat(ai): MiniMax video generation (H3 and Hailuo) (#4309)
* feat(ai): MiniMax video generation (H3 and Hailuo)

Adds a minimax-video-generation provider to the video driver. H3 models go
through MiniMax's v2 task API and are billed per second of output; Hailuo
models go through the v1 API, exchange a file id for a download URL, and
are billed per clip. The shared `minimax` chat key is reused, with the
chat block's `/v1` base URL reduced to its host.

WIDTHxHEIGHT sizes now map onto tiers for catalogs that include a 2k or
4k tier.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

* fix(ai): validate MiniMax video inputs and resolution tiers

---------

Co-authored-by: Claude Fable 5.1 <[email protected]>
2026-10-24 13:49:21 -07:00
Nariman Jelveh 8a55debdae fix(dashboard): sixth adversarial review — bugs, UX, a11y, perf (#4314)
* fix(dashboard): render in the user's saved language

The dashboard never read the account's preferences: only the desktop did,
on boot. Everyone saw it in English whatever language they had chosen,
until the Files tab's Show hidden toggle switched a running session over,
to the browser's language when none was saved, since that was the
fallback the toggle wrote back.

The desktop's preference loading moves into load_user_preferences, which
the dashboard now runs in the background. The first paint uses the
language this browser last loaded, so nothing waits on the read; a saved
language that differs reloads into it, and a changed Show hidden setting
refreshes the open folder. The tab names are read when the sidebar is
drawn rather than when the modules load. A failed read keeps the cached
preferences instead of resetting them to English.

* fix(dashboard): label credit amounts with a key meant for them

Home's allowance and the Usage tab's total and Credits column used the
`credits` key, which the translations render in its acknowledgments sense:
"5,000 Mitwirkende" (contributors) in German, "Авторы" in Russian,
"特别鸣谢" in Chinese. They now use `usage_credits`, which falls back to
English until it is translated, as `monthly_credits` already does.

* fix(dashboard): enable the token and 2FA cards once the email is confirmed

The dashboard had no handler for the `user.email_confirmed` socket event,
and Account and Security pick the token button's state and whether to
offer 2FA when they render. A user who confirmed their email in another
tab, or in this one through the confirmation prompt, kept a disabled
"Create token" and no 2FA card until a reload.

The dashboard now refreshes the user on that event and renders both tabs
again, as it already does after a temporary account is saved.

* fix(dashboard): save only the profile fields that were edited

Edit profile sent both the display name and the bio on every save, taken
from this session's copy of the profile. That copy comes from the cached
user and can be stale (the profile changed on another device) or empty
(the profile read failed), so changing only the bio wrote the old display
name back, and a save from an empty prefill erased the bio.

The dialog now sends just the fields whose value changed from what it
opened with, and closes without a request when nothing did.

* fix(dashboard): don't show storage as 100% while there is room left

Home and Usage rounded the storage share to a whole percent, so an
account at 99.5% or more read "100%" while uploads still fit. The usage
meter already holds at 99% until the allowance is spent; that rule moves
into meterPercent and the storage meters use it too.

* fix(dashboard): keep keyboard focus inside the open phone drawer

Below 768px the sidebar is a modal drawer over a scrim, but opening it
left focus on the menu button, and Tab past its last entry walked into
the dimmed content behind the scrim. Opening it now focuses the current
page's entry, and Tab and Shift+Tab cycle within it while it is open.

* fix(dashboard): tapping into a Files name field edits the name

On a phone or tablet, tapping the name field of a new folder or a rename
(which iOS needs before it shows the keyboard) reached the row's tap
handler, which opened the folder or launched the file and threw the edit
away. A long-press in the field opened the touch sheet instead of
selecting text, and on desktop a right-click in it showed the file menu
instead of the browser's Cut/Copy/Paste.

The row's tap and menu handlers now leave the name field alone.

* fix(dashboard): don't offer Cut on items the user can't move

A file's menu offered Cut on anything that wasn't immutable, and Shared
rows aren't marked immutable. Cutting a file someone shared with you, or
anything inside a read-only shared folder, then pasting it, ended in a
"Forbidden" alert and an emptied clipboard. Cut now needs the same
permission as Delete in that menu, as Ctrl+X and the selection's Cut
already do.

* fix(dashboard): don't paste or upload into the folder being left

Opening a folder clears the listing at once but only moves currentPath
when the new listing arrives. Ctrl+V, a file dropped from the computer
onto the pane, or Paste from the background menu during that wait went
into the folder the user had just left, and since the refresh then
showed the new one, the items seemed to vanish.

Those three now refuse while another folder is loading, as New Folder
already did.

* fix(dashboard): clear the drop highlight when Escape cancels a drag

Escape ends a Files drag with a synthetic mouseup, which jQuery UI
treats as a release outside every target: it fires no `out`, so the
folder row under the pointer kept its dashed drop outline and tint (and
a sidebar folder or breadcrumb its highlight) after the drag was gone.
The Escape branch now takes those off.

* fix(gui): show the "can't rename in Trash" alert without entities

Pressing F2 on a trashed item (on the desktop or in the dashboard's
Files tab) showed "This item can&apos;t be renamed because it&apos;s in
the trash": the message was encoded by i18n and then again by UIAlert.

* fix(gui): show the failed-upload title without entities

The upload progress window writes its error title with innerText, but
was handed the HTML-encoded i18n string, so French showed "Échec de
l&apos;importation des fichiers". The Files tab's two upload paths and
the desktop's now pass the plain string.

* feat(dashboard): open a Files row's menu with the Menu key or Shift+F10

Share, Download, Open With and Properties were reachable only by
pointer: the ⋯ button isn't focusable, and the keyboard's menu keys
fired a contextmenu on <body>, which the listing ignores. Both keys now
open the selection's menu at its row's ⋯ button, where arrow keys and
Enter already work.

* fix(dashboard): let Shift+Arrow shrink a Files selection

Shift+Arrow only ever added the next row, so after Shift+Down twice,
Shift+Up left all three rows selected instead of going back to two. A
run of Shift+Arrows now selects the range from the row it started on to
the cursor, on top of whatever was selected before the run, so moving
back shrinks it. A click or a plain arrow starts a new run.

* fix(dashboard): name folders in the Back/Forward history menus as the crumbs do

Holding Back or Forward listed each folder by its path's basename: `/`
was a blank entry, the Shared view read "shared", Trash stayed in
English, and a trashed folder showed its uid. The menus now use the same
names the breadcrumbs show.

* fix(dashboard): show a shortened Files name in full on hover

Grid tiles and a resized Name column shorten long names in the middle
("Quarterl…draft v2.xlsx"), and nothing showed the rest short of opening
Rename or Properties. A shortened name now carries the full one as its
tooltip.

* fix(dashboard): keep keyboard focus on the Files breadcrumbs

Every listing rebuilds the breadcrumbs, which took a keyboard-focused
crumb with it: pressing Enter on one, or any refresh of the folder while
a crumb had focus, left focus on <body>, and the next Tab started over
from the top of the page. Focus now moves to the same folder's crumb,
or to the one for the folder just opened.

* fix(dashboard): reach Home's Shared and Trash rows from the keyboard

At 480px wide or less (a phone, or a desktop browser zoomed in) the
folder sidebar is hidden and Home lists Shared and Trash as rows instead.
Those rows only had a click handler, so a keyboard user couldn't get to
either. They are now focusable links that open on Enter or Space, with
the sidebar's focus ring.

* fix(dashboard): keep the Files selection across a sort or view change

Sorting by a column, switching between list and grid, or toggling Show
hidden lists the folder again, and the rebuild dropped the selection and
the shift anchor: select three files, sort by Size, and nothing was
selected. Only the refresh on returning to the browser tab carried them
across. That logic is now refreshKeepingSelection, and those three use
it too.

* fix(dashboard): don't drop a folder open during a Files refresh

A refresh of the folder on screen (after an upload or paste, Refresh, a
sort, or returning to the browser tab) set the same "already rendering"
flag that a navigation sets, so double-clicking a folder, a sidebar
folder, a crumb or Back during that round-trip did nothing, with no
spinner to explain it.

Opening another folder now takes over from a refresh in flight; the
refresh notices it was superseded and drops its result. A navigation
during another navigation is still ignored, as before.

* fix(dashboard): keep the Files marquee when an item arrives mid-drag

Adding a row clears the empty-folder notice by removing every non-row
child of the listing, and the selection marquee is one. A file added
from elsewhere (an upload, another tab, an app) while the user was
drawing a selection made the rectangle vanish under the pointer.

* perf(dashboard): don't rewrite unchanged Files names while resizing

Every width change (a window resize, collapsing the sidebar) re-fits the
item names, at most once a frame, and wrote each name's text back even
when it came out the same. Each write invalidated layout for the whole
list. With 300 rows that took ~6.4ms a frame in list view and ~4.9ms in
grid; skipping the unchanged writes brings it to ~0.7ms and ~3.2ms.

* fix(dashboard): tell assistive tech the Files Select toggle's state

The Select button only toggled a class, so a screen reader announced a
plain button with no state; it now carries aria-pressed. Sort and View
open menus and now say so with aria-haspopup.

* fix(dashboard): show a person's strongest grant in the share list

When someone held two grants on one item (the owner gave them view
access, and someone the owner let share gave them edit), the share
dialog showed whichever grant the listing returned first, so their row
could read "Can view" while they could edit. A second grant on the same
item now counts toward the row with the stronger of the two levels.

* fix(dashboard): only offer share controls on grants the user can change

The backend lets an item's owner change or withdraw any share of it,
and anyone else only the grants they issued (or their own access). The
share dialog offered the level select and Remove on every row. Someone
the owner let share who removed a person the owner had added got
"Removed bob" while nothing changed, and lowering their level added a
second grant that changed nothing either.

The access list now records who issued each row's grants. A user who
doesn't own the selection gets the controls only on rows whose grants
they issued all of, plus Remove on their own row to leave; other rows
show their level as a tag.

* fix(dashboard): update Files' shared markers after the share dialog closes

The share dialog marks the Files rows it covers as shared or not from
the listings it reloads after each change, and skipped that whole reload
once it was closed. Closing it while a share or removal was still saving
left the row without its shared marker (or with a stale one) until the
folder was reloaded. The markers are now updated either way; only the
dialog's own list is skipped once it's gone.

* fix(dashboard): mark rows shared after sharing more than 50 at once

The share dialog updates the Files rows' shared markers from the share
listings it reloads after a change, and doesn't reload them for more
than 50 items. Sharing such a selection, with a person or by link,
reported "Shared with bob on 55 items" while no row showed the marker
until the folder was reloaded. Those grants now mark their rows
directly.

* fix(gui): unbind each context menu's mousemove handler with the menu

The submenu-aim code bound a mousemove handler on the document for every
menu and submenu and never removed it. Each one kept its closed menu and
items alive and ran on every mouse move for the rest of the session; ten
right-clicks in Files left five such handlers behind. The handler now
comes off when its menu is removed. Moving diagonally into a submenu
still keeps it open.

* fix(gui): ignore clicks on a context menu's dividers

A divider is an <li> like the items, so a click on one (they're 11px
tall, easy to hit between two items) ran the item handler with no
action behind it and threw "Cannot read properties of undefined". The
item handler now skips dividers; the menu stays open, as a native
menu's separator does.

* fix(dashboard): warn that removing a team removes everyone in it

Removing a team from a single item's share dialog asked "Remove Design's
access to this item?", the wording used for one person. The desktop's
share window adds "Everyone in the team loses it."; the dashboard now
uses that confirmation for teams too.

* fix(dashboard): count folders, not items, in "via N folders"

A person who reaches a multi-item selection through more than one shared
ancestor gets a "via N folders" note in the share dialog, and N was the
number of items reached that way: five files under two shared folders
read "via 5 folders". It is now the number of distinct folders.

* fix(dashboard): don't treat a long tail after a dot as a file extension

Files middle-truncates long names and keeps everything after the last
dot whole as the extension. For "Report v1.2 final draft for the board
meeting" that kept ".2 final draft for the board meeting", so the tile
read "….2 final draft for the board meeting" and lost the start of the
name. Only a short tail without spaces now counts as an extension; this
one shortens to "Report v1.…d meeting".

* fix(dashboard): keep Tab inside the touch context menu

The touch menu sheet is modal, but Tab and Shift+Tab left it for the
page behind the backdrop while it stayed open; closing it then no
longer returned focus, since focus was elsewhere. Tab and Shift+Tab now
move between its items like the arrow keys, wrapping at the ends.

* fix(dashboard): say "the other item" when one item is missing a grant

The share dialog's "Add to all" link told screen readers "Give bob
access to the other 1 items" when the person was missing from a single
item of the selection. That case now has its own singular string.

* fix(dashboard): announce a failed Properties load, on both tabs

When Properties couldn't read the item, the General list showed the
error but stayed aria-busy, so screen readers kept it unannounced, and
the Versions tab stayed blank. The list is no longer busy once the
error shows, and Versions shows the same error.

* perf(dashboard): launch the add-app modal's apps without reloading the grid

"Browse the App Center" and "Build with AI" go through the same intro
as a deep link, which first awaited loadApps(). With the grid already
loaded that started a fresh fetch of every app list and the saved
layout, and the launch waited on it: with an 800ms connection, ~1.35s
before anything opened instead of ~0.1s. The intro now uses the grid
already loaded, and only joins or starts a load when there is none.

* fix(dashboard): don't hide the Apps grid on every search keystroke

Search re-rendered the grid behind the first load's fade-in, which waits
for page one's icons. Each keystroke made fresh <img> elements, so an
icon that was slow or failing (an unreachable icon host) blanked the
whole grid until it settled, about a second per keystroke with a 1s
icon and up to 1.5s if it never answers. Search, its clear button and
entering reorder mode now render the results at once.

* fix(dashboard): stop the Apps pager's dots jumping back mid-flip

A page arrow, a dot, or the wheel starts a smooth scroll and marks the
target page at once, but the scroll handler then read the pages passed
over on the way as the current one. The active dot went forward, back
to the old page, and forward again, and a second click on the arrow
added to the stale page: two quick clicks showed 0 → 1 → 0 → 2 → 1 → 2.

While such a scroll is on its way, the pager (and the open folder's)
now ignores the positions in between, until it arrives or a second has
passed. The user's own scrolling still moves the dots.

* fix(dashboard): keep paging while a dragged app is held at the edge

Holding a dragged tile still at the Apps grid's edge (or an open
folder's) turned one page and stopped: the next dwell only started on
another pointer move, so a mouse held steady stayed on the second page
of three. Once a flip settles, the dwell starts again if the pointer is
still in the edge zone.

* fix(dashboard): don't turn Apps pages on a trackpad pinch

A pinch on a trackpad reaches the page as wheel events with ctrlKey
set, and the Apps grid (and an open folder) read them as a wheel flick
and turned the page while the user zoomed. Those events are now left
alone.

The handlers also called preventDefault() from a listener the app makes
passive, which does nothing and logged a console warning on every wheel
tick; that call is gone, as the scroller has no vertical scroll to stop.

* feat(dashboard): open the first Apps search result on Enter

Typing in the Apps search and pressing Enter did nothing; reaching a
result took ArrowDown and then Enter. Enter in a non-empty search now
opens the first match (or a folder that matches), as launchers do. It
does nothing while an IME composition is open, or when nothing matches.

* fix(dashboard): include an open folder's page dots in its Tab cycle

The open folder keeps Tab inside it, but its list of stops left out the
page dots, which are buttons. Tab never reached them, and once one had
focus (clicking it focuses it) the next Tab walked out of the modal
folder to the page behind. The dots are now part of the cycle.

* feat(dashboard): open an app tile's menu with the Menu key or Shift+F10

An app or folder tile's menu (Open in New Tab, Copy Link, Uninstall,
Remove from Folder; a folder's Rename and Ungroup) opened only on a
right-click or long-press, and browsers don't all turn the keyboard's
menu keys into a contextmenu event (macOS has none). The Menu key and
Shift+F10 on a focused tile now open its menu at the tile, where arrow
keys and Enter work.

* fix(dashboard): keep keyboard focus on the grid after Ungroup

Ungrouping a folder from its menu re-rendered the grid without the
folder's tile, so a keyboard user who had it focused was left on
<body> and the next Tab started from the top of the page. Focus now
moves to the folder's first app, which takes the folder's place.

* fix(dashboard): give links AA contrast in the light theme

Link-colored text (Home's "View more" and "Upgrade", the notification
panel's "Mark all read" and "Retry", Properties' website links, the
token dialog's sessions link) used #5271ff, 4.07:1 on white and 3.97:1
on the content background, under the 4.5:1 AA minimum for text that
size. The light theme's link is now #3d5bd9 (5.7:1), with a darker
hover. Dark mode already passed.

* fix(dashboard): give success text AA contrast in the light theme

Success text (the 2FA "enabled" line, Home's active-plan badge, the
share dialog's confirmations) used #03933a: 4.01:1 on white and 3.80:1
on its green tint, under the 4.5:1 minimum. It is now #15803d (5.02:1
and 4.75:1); the success check icons that share the token darken with
it. Dark mode already passed.

* fix(dashboard): give danger buttons and error text AA contrast

White text on the red fills fell short of 4.5:1: the Apps uninstall
confirm (#e53935, 4.23:1), the delete-account button (#f00808, 4.43:1)
and the Files drag "Cancel" zone (#ef4444, 3.76:1). So did light-theme
danger text on its pink tint (#dc2626 on #fef2f2, 4.41:1), used for
form, share, crop and notification errors and the past-due badge.

The fills are now #d32f2f, #d90707 and #dc2626 (4.8:1 to 5.3:1), each
with a darker hover, and danger text is #b91c1c (5.9:1 on the tint).

* fix(dashboard): give the app feedback dialog's Send button AA contrast

The feedback dialog's primary button (Send, and Close once sent) put
white text on a light-blue gradient, 2.66:1 at the top and 3.42:1 at the
bottom. It now uses the dashboard's primary button colors, which meet
4.5:1 in both themes, like the dialogs around it.

* fix(dashboard): show the drop outline and dwell fill on grid folders

Dragging files over a folder in grid view only tinted it like a
selection: the grid's own border, hover and selected rules out-ranked
the shared drop-target and spring-load rules, and the hover rule's
background shorthand wiped the dwell fill. The folder then opened after
700ms with no sign it was about to. Grid and compact-grid folders now
get the list's dashed outline and filling dwell, with the padding
trimmed to match so the tile doesn't shift.

* fix(dashboard): keep the Files preview and drag ghost above the window

The Space-bar image preview (z-index 9999) and the drag ghost (10000)
are appended to <body>, beside the dashboard window. On a touch-only
tablet with a keyboard every window is pinned to 9999999, so the preview
opened invisibly behind it from the first press; on desktop the
window's z-index climbs with each click and overtakes both in a long
session. Both now sit in the band the sidebar tooltip and the toasts
already use.

* fix(dashboard): give small dashboard controls 44px touch targets

Several controls a phone or tablet user taps were under the 44px
minimum touch size, with nothing enlarging them on a coarse pointer:
the Files toolbar buttons (32px), the touch menu's rows (36px tall),
the drawer's close button (32px), the notifications close (30px), the
Properties close (32px), the feedback dialog's ✕ (30px), the add-app
Back arrow (26px), the Apps modal buttons (34px tall) and the share
dialog's Copy link (38px tall).

Following the earlier touch fixes, the icon buttons keep their look
and get a 44x44 hit area, and the text buttons and menu rows grow to
44px. Mouse and trackpad layouts are unchanged.

* fix(dashboard): keep a landscape phone's notch clear of the dashboard

A phone held sideways and wider than 768px (an iPhone Pro Max, a large
Android) gets the desktop layout, and the page asks for the full screen
with viewport-fit=cover. Nothing in that layout read the side safe-area
insets, so with the notch on the left the sidebar's icons sat under it
(x=24 against a 47px inset), and with it on the right the content's
edge and the toasts did.

The sidebar (open or collapsed) now adds the left inset, and the content
area, including Apps and Files, and the toasts add the right one. The
insets are 0 everywhere else, so other layouts don't change.

* fix(dashboard): size the touch menu and bottom sheets to the visible height

The touch context menu, the app feedback dialog and the profile-photo
crop sheet capped their height in vh. On phones 100vh is the viewport
with the browser bars hidden, so while they show these could be taller
than the screen. The touch menu, which ContextMenu.js keeps inside the
visible height, then ran its last rows off the bottom on a landscape
phone, and the bottom sheets could push their header off the top. They
now cap in dvh, as style.css already does.

* fix(gui): move keyboard focus into the Manage sessions dialog

The sessions dialog is modal but never took focus: opening it from the
keyboard left focus on the button behind the backdrop, and Tab walked
the page underneath instead of the session list. It now focuses its
close button on open, keeps Tab inside, and hands focus back to the
control that opened it when it closes.

* fix(dashboard): return focus after the Security tab's dialogs close

Closing Blocked people, or the 2FA setup or disable dialog opened from
the switch, left keyboard focus on <body>, so the next Tab started at
the top of the page. The 2FA switch also disables itself while its
dialog is up, which drops focus even before then. Focus now goes back
to the button or switch that opened the dialog once it has gone.
2026-10-24 08:30:57 -07:00
Daniel SalazarandReynaldi Chernando f3e8539eaf docs: easy events recipe (PUT-1961) (#3998)
* docs: easy events recipe (PUT-1961)

* docs: catch-up-on-notifications events recipe (PUT-1961)

* docs: notification examples use notif:app-user, which apps can read

* docs: events recipe corrections from a run against a live server (PUT-1961)

* refine and update agents md

---------

Co-authored-by: Reynaldi Chernando <[email protected]>
2026-10-24 15:26:07 +07:00
Nariman Jelveh b3b02b1ebb Fix/dashboard fifth review (#4313)
* fix(dashboard): close the Apps modals and folders only on a press that starts on the backdrop

The uninstall and add-app modals and the open folder closed on any click
whose target was their backdrop. Drag-selecting text in the app request
box and releasing outside the card lands that click on the backdrop, so
the modal and the typed request were thrown away.

The backdrop logic the dashboard dialogs already use moves into
bindBackdropDismiss, and the three Apps overlays use it too.

* fix(dashboard): double-clicking "Add an app" no longer opens App Center

The add-app modal opens under the pointer, so the second click of a
double-click on the "+" tile landed on its first option and launched App
Center (or the AI builder, depending on where the tile sat) without the
user choosing anything. An option now ignores the second click of a
double-click.

* fix(dashboard): act on an app tile once per double-click

A tile acted on every click, so a double-click opened an external app's
site in two browser tabs. Double-clicking an option in the add-app modal
also launched whichever tile sat under the pointer once the modal had
closed (App Center and Recorder together, say). A tile now ignores the
second click of a double-click; the first already launched, focused or
opened what was asked for.

* fix(gui): show app launch refusals in a Puter alert, translated

launchApp reported "You don't have access to …" and "Couldn't open …"
through window.UIAlert, which nothing defines, so both fell through to
the browser's native alert(): a blocking, unstyled, English-only dialog
over the dashboard or desktop. They now use UIAlert directly, with the
text in en.js, and pass the app title unencoded since UIAlert encodes
its message itself.

* fix(dashboard): clicking a toast with nothing to open keeps apps on screen

A toast click brings the dashboard forward before acting on the
notification, hiding every app window. Only share notifications lead
anywhere, so clicking an "events subscription ended" toast (or any other
type) to dismiss it minimized the app the user was working in and showed
a dashboard with nothing to look at. Those toasts now just mark the
notification read.

* fix(dashboard): show an emptied Trash as empty

Emptying the Trash from the Files tab took its rows away but left the
footer on "N items · X MB" and the pane blank, without the empty-folder
notice; the selection bar kept offering actions on rows that were gone.
Emptied from another tab, the listing was cleared with .empty(), which
also dropped the loading overlay and likewise never updated the footer.

Both paths now finish with the empty notice and a fresh footer.

* fix(dashboard): say a folder is empty when its last item goes elsewhere

When another tab, device or app deleted or moved away the last item in
the folder Files was showing, its row faded out and left a blank pane:
the item.removed and item.moved handlers updated the footer but never put
the "folder is empty" notice up. They now do, for rows that were in the
Files listing.

* fix(dashboard): list failed drag-and-drop uploads one per line

The upload-button path was fixed to put each failed item on its own line,
but files dropped onto the listing still joined them with "\n", which
UIAlert's HTML collapses into spaces: every failure ran together on one
line. Both paths now build the message in failedUploadsMessage.

* fix(dashboard): dropping items back into their own folder does nothing

Dropping a selection on the sidebar entry for the folder it came from, or
spring-loading into a folder and coming back to release in the listing,
asked move_items to move every item where it already was, and each one
stopped on a "Cannot move item to its current location" alert in turn.
The breadcrumbs already ignore such a drop; moveDraggedRows now leaves
those rows out too, and a drop with nothing left to move does nothing.

* fix(dashboard): a Ctrl- or Alt-drop on the Trash trashes the items

Ctrl-dragging a file onto the sidebar Trash (or a Trash breadcrumb)
copied it into the Trash: the original stayed put, and the copy carried
no original path, so it was offered Restore and Restore failed. Alt-drops
made shortcuts there the same way.

The four drop handlers' copy/shortcut/move choice moves into
dropRowsInto, which only ever trashes what lands on the Trash.

* fix(dashboard): the first row click after a marquee released outside Files counts

A marquee sets a flag so the click that ends it doesn't clear the new
selection; the listing's click handlers consume it. Released over the
sidebar or outside the window, that click lands elsewhere, the flag
stayed set, and it swallowed the user's next click on a row. The flag now
lapses once the release's own click has had its turn.

* fix(dashboard): Back and Forward follow a folder the user renamed

Open a folder, go back, rename it, then press Forward: history still
held the old path, so Files showed "This folder couldn't be opened", and
the Back/Forward long-press menus listed the old name. A folder renamed
in Files now rebases the history entries in and under it, using the
same path rebase the username-change handler does.

* fix(dashboard): a folder that can't be opened becomes the current folder

When a folder failed to list (deleted elsewhere, share revoked, offline),
Files said "This folder couldn't be opened" but stayed on the previous
folder underneath: history had moved on while the breadcrumbs, sidebar
highlight and New Folder/Upload/Paste still pointed at the old folder,
so a new folder or upload landed somewhere other than where the user
seemed to be.

The tab now moves to the folder it was asked for before showing the
message, and creates nothing in it until it lists again.

* fix(dashboard): double-clicking Save in the photo crop uploads it once

Save closes the crop dialog, but the dialog stays in the page for its
200ms fade-out with its handlers live, so the second click of a
double-click ran Save again and sent the profile picture to the server
twice. Save does nothing once the dialog has closed.
2026-10-23 19:04:36 -07:00
Toshit ChawdaandClaude Opus 5 bdc9aa9e93 Revamp networking api with epoxy try 2 (#3476)
* revamp networking api with epoxy

* update to new epoxy build

* force wisp auth

* emit Error objects from socket 'error' events

main tightened the socket error contract to pass an Error (see
types/modules/networking.d.ts, which documents the reason as
`error.message`); the epoxy rewrite still emitted a bare string,
matching the older `(reason: string)` declaration.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

* test: cover the epoxy-based networking client

Replaces the coverage lost when the wisp implementation was removed:
PSocket.test.js and PWispHandler.test.js were bound to PWispHandler,
parsers.js, and wispInfo, none of which survive the epoxy rewrite.

  PSocket.test.js  - connect/retry, inbound data, write normalisation,
                     close, and the TLS event remapping. Driven through
                     real ReadableStream/WritableStream pairs so
                     reader locking, cancel, and abort behave as they
                     do in the browser.
  index.test.js    - relay-token exchange (auth header, 401 re-auth and
                     retry, malformed responses) and the epoxy client
                     cache (keying, refresh, in-flight sharing).
  requests.test.js - pFetch delegation, cache invalidation, and the
                     api call logger's request description.
  epoxy.test.js    - the hand-packed wisp password extension payload.

The wasm runtime loader is left to integration coverage; it needs a
network fetch and a browser. createPuterPasswordBuilder is exported so
the byte layout can be tested against an injected fake runtime.

One skipped test records a pre-existing defect: on a failed write,
#readLoop's normal-termination path races #closeStreams and emits
close(false) after an error.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

* fix: report socket close flags and epoxy init failures accurately

PSocket #readLoop raced the teardown it was being torn down by. A failed
write sets #closing and hands the close event to #closeStreams, which has
to await reader.cancel() and writer.close() before emitting close(true).
That cancel resolves the loop's pending read with {done: true}, so the
loop broke, reached its own #emitClose(false), and set #closed first --
callers were told the socket shut down cleanly right after an error.
The loop now defers the close event whenever a teardown is under way, in
both its normal and its throwing path, and #closeStreams always emits.

getEpoxyClient swallowed every init failure and resolved undefined, so a
dead relay reached callers as "cannot read properties of undefined
(reading 'connect')" instead of its cause. It now re-throws, still
without caching the failed attempt. A socket reports "wasm unavailable"
or "Failed to create relay token (HTTP 503 ...)" and closes with the
error flag set.

Two related cache faults fell out of that rewrite: the in-flight
early-return ignored `refresh`, handing a caller the very attempt it
asked to replace, and it ignored the cache key, so an attempt started
before a token change satisfied a request made after it. Reuse is now
conditional on both.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

* restore the public API surface dropped in the epoxy revamp

The epoxy rewrite kept the shape of the old networking client but lost
three things callers depend on:

- JSDoc on the exposed methods. Re-added on PSocket (constructor, on,
  addListener, write, close), PTLSSocket and pFetch, along with the
  SocketEvent typedef and a Networking type on netAPI. `on` carries the
  TLS event aliasing docs that used to live on PTLSSocket, since that
  behaviour moved here.

- socket.write's error message, which lost its trailing '!!'. Apps match
  on the message, so it is restored verbatim.

- pFetch's client-side validation. Epoxy reports a bad request as an
  opaque wasm value from deep inside the transport, so the scheme check,
  the Content-Length/body check and the malformed-request TypeError are
  made before a relay is dialled, rejecting with exactly what they
  rejected with before. Validation builds its Request from a clone, or it
  would consume the body epoxy is about to send.

The latter two are what the net API suite asserts on node, browser and
workerd, and it has been failing on all three since the revamp.

addListener takes named parameters so it can be typed, and three unit
tests that pinned pFetch's habit of forwarding unvalidatable input were
updated to the documented behaviour.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

* update epoxy client to 04e4930

connectionPrefs changed shape: the handshake and its required extension
list are one object now rather than a tuple, and handing the new build
the old form throws part way through the handshake.

The bump is not optional. The previous build cannot reach the current
relay at all: it enforces the required puter password extension even when
a relay answers with a v1 CONTINUE, and fails with "Protocol extensions
not supported" -- which is what puter.cafe answers with today.

Verified against a local relay: fetch, POST bodies, https, raw sockets and
TLS sockets, the last exercising connectTls, whose third parameter became
an options object in this build.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

* bump epoxy

* dont require ext

* chore(puter.js): update Epoxy runtime to 68c6904

* chore(puter.js): update Epoxy runtime to affd9f5

---------

Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
2026-10-23 16:57:25 -07:00
Nariman Jelveh 21c1e4142e fix(dashboard): fourth adversarial review — bugs, UX, a11y, perf (#4312)
* fix(dashboard): keep Escape working when two share or properties dialogs are open

Each share and properties dialog bound its document keydown under one fixed
namespace, so closing either instance unbound every instance's Escape.
Double-clicking Share… or Properties in the context menu opens two; after
the top one closed, Escape no longer closed the one left behind. Each dialog
now binds under its own namespace.

* fix(dashboard): don't close Properties when a text selection ends on the backdrop

Drag-selecting the path or UID in the properties dialog and releasing over
the dimmed backdrop closed the dialog: the click lands on the overlay when
press and release are on different elements. The backdrop now closes it
only when the press started there, as the share dialog already does.

* fix(dashboard): keep focus in Properties after disassociating one of several websites

With more than one website on an item, disassociating one removed the
focused button and returned early before refocusing the dialog, so focus
fell to the page and the next Tab landed in the dashboard behind the modal.
The dialog takes focus back as soon as the entry is removed.

* fix(dashboard): handle the Properties stat failure instead of leaving it unhandled

puter.fs.stat both calls the error callback and rejects. The properties
dialog rendered the error from the callback but dropped the promise, so
opening it on a deleted item or offline logged an unhandled rejection.

* fix(dashboard): don't double-encode the "anyone" and withheld-invite share rows

The share dialog encodes row names when it renders them, but these two
labels came from i18n() already HTML-encoded, so a translation containing
an apostrophe or ampersand would show its entity text. They're now taken
unencoded like every other row name.

* fix(dashboard): make the touch context menu reachable by assistive tech and keyboard

The touch action sheet had no roles and never took focus: it was appended
at the end of the page, so a screen reader announced nothing on open and
focus stayed on the page behind it. It's now a menu of menuitems (with
separators and submenu popups marked), takes focus on open, gives it back
on close, moves between enabled items with the arrow keys and Home/End,
and shows an inset focus ring for keyboard use. Taps draw no ring.

* fix(dashboard): keep Apps folders and order when a settings read fails

The Apps tab read the saved order, folders and uninstalled-apps list with
`.catch(() => null)`, so a failed read looked like a user who had never
saved any. One flaky request emptied the folders and reset the grid to
the default order, and the next drag or folder edit wrote that loss back
to kv for good.

A failed read now keeps what the session already has (the last
uninstalled-apps list included). Until a read of the order or folders has
succeeded, saves stay in memory instead of overwriting the stored value.

* fix(dashboard): drop the Apps tile a failed deep-link launch drew

Landing on /app/<name> (or choosing an option in the add-app modal) adds
the app's tile before the launch runs, because the launch is what installs
it. When the launch failed — no access, no app token, a thrown error — the
tile stayed for the rest of the session, survived refreshes, and every
click on it failed again.

The launch outcome is now passed to settleDeepLinkLaunch, which takes a
tile it drew back out (and restores a previous uninstall record) when
nothing was launched.

* fix(dashboard): put a re-installed app back in its saved slot

Uninstall keeps the app's name in the saved order so a re-install returns
it to its place, but a deep-link landing appended the tile at the end of
the grid. The install animation played on the last page and the tile then
jumped back to its saved slot on the next refresh. The landing now places
the tile through reconcileAppOrder like every other load.

* fix(dashboard): don't double-encode the add-app launch failure alert

UIAlert encodes its message, so passing it the already-encoded i18n string
showed entities on screen in languages with an apostrophe (French read
"s&apos;est mal passé").

* fix(dashboard): say so when an app tile fails to launch

A tile click whose launch rejected (for example the app lookup failing)
only logged to the console: the icon flourish played, settled, and nothing
opened. It now shows the same alert the add-app modal shows for that
failure.

* fix(dashboard): keep focus inside the Apps modals and return it on close

The uninstall and add-app modals are aria-modal but had no Tab trap, so
Tab walked into the search box and tiles behind the scrim, where typing
filtered the grid under the modal. Closing the add-app modal, and
confirming an uninstall, dropped focus to the page body.

Tab now cycles inside both cards; closing the add-app modal returns focus
to the add tile, and an uninstall leaves focus on the tile that takes the
removed app's slot.

* fix(dashboard): upload from a folder's menu into that folder

Upload in a sidebar or breadcrumb folder's menu opened the shared file
picker, whose handler always uploaded into the folder on screen: files
picked for Pictures while viewing Documents landed in Documents. The menu
now hands its folder to the picker.

* fix(dashboard): no New, Upload or Paste where nothing can be created

New Folder, Upload, the New/Paste/Upload menu items and file drops were
only refused in the Trash itself and the Shared view. Inside a trashed
folder they created items without the metadata Restore needs, and at /
every create failed (an upload there threw before its progress window
existed and showed nothing). One check now covers the Trash, folders in
it, the Shared view and /.

* fix(dashboard): make Paste Into Folder work after a cut

The row menu's Paste Into Folder handed a cut to the desktop's
move_clipboard_items, which finds items by their on-screen elements and
plain-string paths. The Files tab's cut stores {path, uid} objects, so
nothing moved and the clipboard was emptied. The menu now takes a paste
handler, and Paste, Ctrl+V and Paste Into Folder share one.

* fix(dashboard): don't cut or copy trashed items out of the Trash

The single-item menu still offered Cut on trashed items, and Ctrl+C
copied them; pasted elsewhere, they kept their raw name in the Trash, a
uid. Trashed items leave through Restore, which puts the original name
back.

* fix(dashboard): Delete Permanently works in every language

The confirmation compared the clicked button's label with 'Delete', so in
any other locale confirming did nothing. The buttons carry values now.

* fix(dashboard): refresh Files on return after a cancelled rename

Escape, or keeping a name unchanged, hid the rename editor but left its
-active class, and the refresh on returning to the browser tab takes that
class for an open editor: the listing stopped refreshing until the next
navigation.

* fix(dashboard): clear a drop target's highlight after the drop

jQuery UI fires no `out` on drop, so the folder row a selection was
dropped on stayed selected (and went to the Trash with the next Delete),
a sidebar folder stayed highlighted as the current one, and a breadcrumb
kept its drop outline.

* fix(dashboard): listing drops keep working after a drop on a folder row

Dropping files from the computer onto a folder row stopped the native
event before the listing's dragster saw it, leaving its enter state stuck:
later drags didn't highlight the listing, and drops on its background
uploaded nothing.

* fix(dashboard): offer Restore only for items Restore can handle

Selecting files inside a trashed folder offered Restore in the
multi-select menu and the selection bar, which then failed once per item:
only direct children of Trash carry restore metadata. The trashed
multi-select menu also drew two separators in a row.

* fix(dashboard): show names as typed in conflict and move-failure alerts

These alerts html-encoded the file name before UIAlert encoded the
message again, so "R&D.txt" read "R&amp;D.txt". "already exists." is
translatable now too.

* fix(dashboard): list failed uploads one per line

The failed-upload list was joined with newlines, which the alert collapses
onto a single line.

* fix(dashboard): translate the shortcut badge's tooltip

* fix(dashboard): select mode toggles rows with any pointer

In select mode, a stylus or mouse tap on a phone or tablet toggled the row
in pointerdown and again in click, leaving it as it was; in a narrow
desktop window a click replaced the selection and the selection bar never
showed. Select mode now toggles in click for every pointer and shows its
bar whenever it is on.

* fix(dashboard): show items added elsewhere while New › Web Link prompts

The New menu's create wrapper suppressed item.added for as long as it ran,
which for Web Link includes its URL prompt, so files added to the folder
meanwhile never appeared. It now counts only from the actual create. A
create that fails now says so instead of failing silently.

* fix(dashboard): keep the Files spinner under the mobile drawer

The loading overlay sat at the maximum z-index, so on phones it drew over
the open drawer and its scrim and took their taps while a folder loaded.
It only needs to sit above the rows.

* fix(dashboard): don't show a failed usage request as zero usage

getMonthlyUsage() resolves with the error body when the request fails, and
Usage and Home read it as a month with nothing used: $0.00 of $0.00, 0%,
and on Usage an empty table replacing the one already on screen. A body
without a usage report is treated as the failure it is, so the last good
numbers stay and a first load shows the "unavailable" note.

* fix(dashboard): draw Files select-mode accents in the selection color

The checked checkbox, the active Select button and the Done button were
filled with --primary-color, the desktop's text color rather than an
accent: charcoal by default, and white (on white text and ticks) for
anyone whose desktop theme uses light text. They use --select-color, the
accent the rest of the selection already uses.

* fix(dashboard): stop the mobile drawer animating under reduced motion

The reduced-motion override for the drawer, its scrim and the menu button
came before the drawer rules it was meant to cancel, and the open
drawer's rule outranked it too, so the drawer still slid and the scrim
still faded. The override now follows those rules and covers the open
state.

* fix(dashboard): make the toast close button readable in dark mode

Dark mode gave the close button the card color and then inverted the
whole element, which turned the disc light grey and the cross white:
about 1.4:1. Without the background the invert gives the light-mode
button's mirror, a dark disc with a white cross.

* fix(dashboard): leave the temporary-account state once the account is saved

Saving a temporary account from the user menu promoted the user in place
without a reload, but Home kept its Save session warning (reopening the
save dialog for a saved account) and Account and Security stayed without
their password and email cards until the page was reloaded. The dashboard
now reacts to account-saved, which fires after the user is updated.

* fix(dashboard): give the API token field 16px text on touch screens

The coarse-pointer rule that sets the token field to 16px, as the other
dialog inputs are, came before the field's own 13px rule of the same
specificity, so the 13px won and iOS zoomed the page when the field was
focused. The touch rule now follows it.

* fix(dashboard): lighten the General access dropdown arrows in dark mode

The dark-mode chevron override covered the share modal's recipient
selects but not the two General access ones, which kept the darker
light-theme stroke on the dark card.

* fix(dashboard): don't leave the sidebar collapse button shaded after a tap

Its hover style was the sidebar's only one not limited to devices that
hover, so on a wide touch screen (an iPad in landscape) the button kept
its hover shade after being tapped.

* fix(dashboard): refresh the Usage tab when the subscription changes

An upgrade from the Usage tab's own button, or a return from the billing
portal, refreshed Home behind it while the Usage meter and plan button
kept the old allowance until the user left the tab and came back. Usage
now handles puter:subscription:changed while it is showing.

* perf(dashboard): load usage and storage once when the dashboard opens

Home and Usage both load the monthly usage and the storage space when the
dashboard opens, whichever tab it opens on, so every open sent
/metering/usage and /df twice. Overlapping loads now share one request.
Nothing is cached: a load after the shared one settles fetches again.

* fix(dashboard): keep keyboard focus on the Usage table's Show more/less

The table redraws without the button that was pressed, so focus fell back
to the page and the next Tab started from the top. Focus now moves to the
button that replaces it, as the sort headers already do.

* fix(dashboard): give primary buttons readable contrast

The dialogs' primary buttons, the share modal's Share and the photo
crop's Save were filled with the link color under white text: 4.07:1 in
light mode, 3.08:1 in dark, where hover lightened it further to 2.38:1.
They take their own fill now, #4a67f5 (4.62:1) darkening to #3d5bd9
(5.68:1) on hover in both themes; links keep their color.

* fix(dashboard): keep collapsed-sidebar labels above the dashboard

The hover label is attached to body at z-index 1000, while the dashboard
window's z-index climbs with every click (focusWindow raises it). Over a
long session the window passed 1000 and the labels rendered behind it,
so hovering the collapsed sidebar showed nothing.

* fix(dashboard): bring muted and tertiary text up to AA contrast

The muted and tertiary greys carry real information (form hints, file
dates and sizes, share "via" lines, the sidebar's team name,
notification text and times) but fell short of 4.5:1: 2.56:1 and 3.54:1
on white, 3.18:1 and 3.13:1 on dark cards. They now clear it on the
cards and the sidebar in both themes (light #5f6b7d / #707070, dark
#8b95a5 / #94949c). Disabled buttons keep the fainter grey through their
own token, so they still read as disabled.

* fix(dashboard): name the plan on self-hosted installs instead of "--"

A self-hosted install has no /marketplace/subscriptions/current route, and
Home treated its 404 like any failed read, leaving "Your Plan" at "--" for
good. A 404 now shows the plan every account starts on. The upgrade badge
and link, which have nothing to reach there, go too, unless the hosted
upgrade dialog is loaded.

* fix(dashboard): make hover and press feedback visible in dark mode

Dark mode's hover color, #2a2a2a, was 1.05:1 against the #262626 cards
and #252525 sidebar, so hovering a notification or sidebar item and
pressing a touch-menu or drawer item showed next to nothing. #333 sits
as far off them as light mode's hover sits off white, and stays below
the selected item's #3d3d3d.

* fix(dashboard): follow a username change in the Files tab

Changing the username, or saving a temporary account under a chosen
name, moves the home folder. The GUI rewrote data-path attributes and the
path globals, but the Files tab kept its current folder, back/forward
history, clipboard and drawn rows on the old path, so sorting, New
Folder, Upload, Up, Back and the breadcrumbs all failed with "This folder
couldn't be opened" until a reload. update_username_in_gui now announces
the change, and the Files tab moves its paths over and redraws.

* perf(dashboard): lay out only the new row when an item is added to Files

Every item added to the open folder from elsewhere (another tab, an
upload, a move into it) restyled and re-truncated every row in the
listing, so a burst of adds was quadratic in the folder's size: 100 adds
into a 300-item folder took ~800ms of main-thread work. The added row now
takes the listing's column layout on its own (~130ms for the same burst).

* fix(dashboard): open the Sort and View menus at their buttons from the keyboard

Enter or Space on either button fires a click with no coordinates, so the
menu opened at the top-left corner of the screen, far from the button.
A keyboard press now opens it below the button; a pointer click still
opens it at the pointer.

* fix(dashboard): open a breadcrumb's long-press menu as the touch sheet

A breadcrumb's long-press opened the desktop context menu at the event's
pointer position, which a long-press doesn't have, so on tablets the menu
landed wherever an unset position put it. Breadcrumbs now open the same
touch sheet, anchored to the crumb, as the sidebar folders and rows do.

* fix(dashboard): don't drag-move items the user can't move

Delete, Cut and their buttons already leave out shared roots and items in
a read-only share, but dropping them on a folder, a sidebar entry or a
breadcrumb still sent the moves, and each came back as a "Forbidden"
alert. Drag moves now leave those rows out too; copying them still works.

* fix(dashboard): clear "1 item selected" when a double-click opens a file

Opening a file by double-click deselects its row, but the footer kept
saying "1 item selected" with nothing selected.

* fix(dashboard): only the Trash gets the Trash label in breadcrumbs

Any folder named "Trash" was shown with the translated Trash label in the
breadcrumbs, so in German ~/Documents/Trash read as "Papierkorb". The
label now goes on the user's Trash folder only.

* fix(dashboard): name a trashed folder by its old name in the breadcrumbs

A trashed item's name on disk is its uid, so opening a trashed folder
showed a crumb like "ffaff8cb-dcaf-44f7-…" while its row in the Trash
read "Old project". The crumb now uses the name the row shows.

* fix(dashboard): show why Delete Permanently failed and keep the item

delete_item fades the item out before the request; when the request
failed it left the item hidden, as if deleted, and passed the alert the
undefined `responseText` of a puter.js error, which threw. The item now
comes back and the alert shows the error's message.

* fix(dashboard): translate the New menu's Web Link and templates strings

"Web Link", its URL prompt, validation and failure messages, the "User
templates" submenu, and the account prompt before publishing a worker
were hardcoded English in the Files New and item menus.

* fix(gui): stop leaking a document click handler per context menu

Every context menu bound its item-click handler on document, delegated
to its own id, and never unbound it. Each menu opened therefore left a
handler that ran on every later click in the page and kept the menu's
items alive, rows and fs entries included. The handler is now bound on
the menu itself and goes with it. It still returns false, so item clicks
keep not reaching document-level click handlers, as before.

* fix(gui): run a context menu item once on a double-click

The menu stays clickable during its 200ms fade-out, so the second click
of a double-click ran the item again: double-clicking Share… or
Properties in the dashboard opened two stacked dialogs. A menu that has
started closing now ignores item clicks.

* fix(dashboard): drag a select-mode selection from its own rows

Select mode returned from pointerdown before recording whether the
pressed row was already selected, so a drag started from that stale
value: dragging selected rows didn't move them, and a stale "selected"
could let a drag from an unselected row clear the selection. The value is
recorded first now.

* fix(dashboard): leave Files select mode when the window widens past 480px

Select mode's Select and Done buttons only show up to 480px, so turning
it on in a narrow window and widening it left every click toggling rows,
with no way out short of narrowing the window again.
2026-10-23 15:24:02 -07:00
Neal Shah 03e483e372 PUT-2252 (#4311) 2026-10-23 17:33:59 -04:00
Nariman Jelveh 9c90c7afde fix(dashboard): third adversarial review — bugs, UX, a11y, perf (#4310)
* chore(dashboard): add the standard file header where it was missing

* fix(dashboard): rename editor keeps entity-like names; rows follow remote renames

The name editor decoded its already-decoded value a second time, so opening
it on a file such as "R&amp;D.txt" and clicking away renamed it to "R&D.txt".

A rename or update from another tab only patched a row's attributes, while
its open/copy/share handlers kept using the entry they were rendered with;
the row now keeps that entry current and refits its name.

* fix(dashboard): Files selection, preview and Trash/Shared edge cases

- A marquee drag without modifiers now replaces a selection made by click,
  Ctrl+A or the arrow keys instead of adding to it.
- With the image preview open, clicking another image switches to it, and
  moving onto a non-image closes it instead of keeping the old image up.
- Cut (Ctrl+X, multi-select menu) skips trashed items, which leave the Trash
  through Restore; Ctrl+V refuses the Trash and the Shared view like the
  Paste menu item does.
- Drops on the Shared sidebar entry or breadcrumb are refused (the drag
  snaps back) instead of sending writes to a path that isn't a folder.
- The Trash selection bar shows Delete permanently again after a selection
  elsewhere had hidden it.

* fix(dashboard): keep an open rename across refreshes; re-sort renamed rows

A same-folder refresh (an upload or paste finishing, a sort change) rebuilt
every row and silently dropped a rename in progress; the editor now comes
back on the same item with the text and caret the user left.

A renamed row whose new name sorts last stayed where its old name was.

* fix(dashboard): Files columns follow the room the listing has

- Resizing a column past the listing's width clamps it instead of resetting
  every column to the defaults; a window too narrow for the saved widths
  lays out at the defaults but keeps the saved widths for when it widens.
- Columns and grid names re-fit when the window or sidebar resizes the
  listing, not only on the next navigation, and grid names hidden behind
  another tab are fitted when the Files tab is shown.
- Names are measured in the font they render in (the compact grid's is
  smaller), and a column drag relayouts at most once a frame.

* fix(dashboard): Files keyboard and touch edge cases

- The marquee selects file rows only, not the Shared/Trash place rows Home
  shows on narrow screens.
- Row shortcuts stand aside while the touch action sheet is open.
- Breadcrumb long-press opens its menu on iPadOS too.
- A folder rendered before the saved view preferences load uses the
  defaults instead of throwing and blocking navigation.

* fix(dashboard): share and properties modals — focus, Escape, link state

Share modal:
- Escape pressed in a window opened over the modal (a verification or
  upgrade gate) no longer closes the modal underneath.
- Turning the link on or off for a selection too large to list now shows
  the result instead of leaving the control on its old state.
- Opening the link up starts at view access; it no longer reuses a hidden
  mode left over from earlier, which could hand out edit access.
- Focus returns to the select the user changed.

Properties modal:
- Takes focus when it opens, keeps Tab inside, gives focus back on close,
  and leaves Escape to a window opened over it.
- Shows a loading state, and opening Versions on a folder before its
  details load no longer leaves an empty body.
- Disassociate is a real button, ignores repeat clicks while pending,
  reports a failure, and keeps the website badge while other sites remain.
- Tabs carry tab/tabpanel roles; transitions respect reduced motion.

* fix(dashboard): account dialogs, avatar saves and usage figures

- A form dialog ignores Escape and backdrop clicks while its request runs,
  so the outcome isn't lost behind a closed dialog; the close button still
  works.
- A new API token, shown only once, survives a stray click outside the card.
- Copy token works on origins without navigator.clipboard (plain-http
  self-hosts) instead of throwing, falling back to a selected field.
- Saving one profile picture and then another before the first finished
  can no longer leave the first one's picture or failure on screen.
- The storage host bar is clamped to the room left beside Puter's share and
  hides its label when there is no host figure.
- Sub-cent dollar costs read "<$0.01" rather than "$0.00", and the usage
  meter reads 99% until the allowance is actually spent.

* fix(dashboard): Apps folder drag, search, keys and loading

- A drag inside an open folder that is cancelled (Escape, blur) or dropped
  back in place puts the tiles back in their saved order; the cached panel
  markup used to skip the rebuild, and the next drag then saved the
  cancelled move too.
- Search no longer matches substrings of generated app ids, so "app" or a
  single hex letter stops returning every website shortcut.
- Arrow keys and Enter stay with the notifications panel, an account dialog
  or the touch action sheet while one is open over the grid.
- The grid shows after 1.5s even if an icon host never answers.
- Refreshing a grid already on screen keeps the page and skips the fade.
- Rename from a folder's menu focuses the name on touch devices too, and
  the folder dialog's label follows renames and counts the apps shown.
- i18n inserts %% values literally, so a name containing $& or $' no longer
  comes out mangled.

* fix(dashboard): shell navigation, a11y and notification sync

- Missed notifications replayed on load toast as one burst (a few, then a
  summary) instead of one toast each.
- A notification read in another tab while this tab's first listing is in
  flight is no longer shown as unread when that listing lands.
- Focus stays in the notifications panel when Mark all read or Retry hides
  the button that had it.
- The active sidebar tab carries aria-current, and the account button is
  named by the username even while the collapsed sidebar hides it.
- Landing with ?ref= replaces the history entry, so one Back leaves.
- Ctrl/Cmd/Shift-click on a sidebar tab opens it in a new browser tab.
- Recently used keeps the newest list when refreshes overlap, and app names
  are URL-encoded in the links it opens.

* fix(dashboard): focus rings, dark-mode contrast and reduced motion

- App tiles show a focus ring, not only the hover enlargement; Home card
  headers get an inset ring the card's rounded clip can't cut off; the
  Delete account button gets the settings cards' ring.
- Pager arrows are mouse-only (the dots page by keyboard), so invisible
  arrows are no longer Tab stops, and an arrow with no page leaves the
  accessibility tree.
- Dark mode: the Files toolbar's action icons match the nav arrows and stay
  readable on the hover chip, hovered breadcrumbs keep their contrast, and
  the shared-item dot's ring follows the row background.
- The row '⋯' button is visible at rest on touch screens.
- A long username or email wraps inside the profile card.
- The touch action sheet, the Files selection bar and the app feedback
  modal stop scaling/sliding under reduced motion.

* fix(gui): desktop rows store remote renames raw; column resize stays in bounds

- The desktop's item.renamed/item.updated handlers wrote HTML-escaped names
  into data-name, the title and the name editor (and titled windows by an
  undefined uid). They store the raw name now, as rename_file does, so the
  editor shows the real name now that it no longer decodes a second time.
- Resizing a Files column while saved widths are too wide for the listing
  starts from the widths on screen, and a width is held at its column's
  minimum, so a drag or auto-fit can't save a negative width.
- Double-clicking a row closes the image preview, which would otherwise
  float above the app window it opens.
2026-10-23 12:32:33 -07:00
Rasad RegmiandDaniel Salazar acd3df8c1b fix(puter-js): replay the request after a successful token_auth_failed reauth (#3954)
* fix(puter-js): replay the request after a token_auth_failed reauth

A 401 carrying the legacy `token_auth_failed` code opened the sign-in
prompt and then dropped the re-authenticated token on the floor:
`resolveReauth()` fell through to `return null` instead of returning
`{ action: 'replay' }`, so the original call was never re-issued and
rejected with `Unauthorized` even though the new session was valid.

Return the replay verdict the `reauth_required` branch already returns.
Both replay paths stay one-shot (`ctx.done.add('reauth')`,
`spec._replayed`), and the auth gate rejects before the route handler
runs, so a replay has no side effects.

Closes #3953

* fix(puter-js): sign in once when a replayed XHR is refused again

handle_resp ran the reauth flow on the replay's own 401, so a
token_auth_failed that survived the replay reset the fresh token and
opened a second sign-in before rejecting. Treat a replay's 401 as final,
as the 403 verification-gate branch already does.

---------

Co-authored-by: Daniel Salazar <[email protected]>
2026-10-23 00:07:56 -07:00
Daniel Salazar 4729d733bb feat(aws): DynamoDB, Polly and Textract can use the default credential chain (#4307)
`useCredentialChain: true` in `dynamo.aws`, `providers['aws-polly']` or
`providers['aws-textract']` takes credentials from the default AWS chain
instead of static keys, as `s3.s3Config` already does. One provider is shared
and refreshes temporary credentials itself.
2026-10-22 21:31:23 -07:00
Nariman Jelveh 26d843778e feat(dashboard): account dialogs as dashboard modals instead of windows (#4305)
* feat(dashboard): account dialogs as dashboard modals instead of windows

Edit profile, change username, change password, change email, create API
token, and delete account in the Account tab (and change password in the
Security tab) now open as dashboard overlay cards, like the share,
properties, and confirm dialogs, instead of desktop UIWindows. They are
centered cards on desktop and tablet and bottom sheets on phones.

- dashboardOverlay.js: the overlay shell UIDashboardDialog already had
  (reveal, Esc/backdrop/close dismissal, focus trap, focus return), pulled
  out so the new form dialog shares it. It also pins the overlay to the
  visual viewport so iOS's keyboard can't cover a sheet's fields, and
  leaves Esc to a modal opened on top of it.
- UIDashboardFormDialog.js: labelled fields, password show/hide, an
  aria-live status line that marks and focuses the offending field, a busy
  submit button, and a done state.
- postUserProtected.js: the cookie-retry / provider re-sign-in / reauth
  flow the four account windows each carried, in one place.
- Deleting an account is one dialog now (warning plus confirmation) rather
  than two stacked windows.
- Touch: 44px controls, 16px inputs (no iOS focus zoom), no autofocus
  that pops the keyboard mid-animation, form buttons side by side on
  phones, safe-area padding dropped while the keyboard is up.
- The replaced windows had no other callers and are removed with their
  tests, which are ported to the new modules.

* fix(dashboard): a double-click that opens a dialog no longer dismisses it

The second press of the double-click lands on the freshly mounted backdrop
and closed the dialog straight away. Backdrop presses with a click count
above one no longer count toward dismissal.

* fix(dashboard): a failed form submit keeps focus in the dialog

Locking the fields disabled the focused control, which drops focus to the
page, so after an error Enter no longer retried and screen readers lost
their place. The control that had focus gets it back once the fields
unlock, unless the handler moved focus elsewhere in the dialog.

* fix(dashboard): a form error keeps the field it names in view

The status line scrolled the dialog body to the bottom, which on a short
screen (a phone with its keyboard up) left the field being focused and
marked invalid scrolled off the top. The body now scrolls back up just
far enough to show that field.

* fix(gui): a blocked re-sign-in popup fails instead of waiting forever

openRevalidatePopup polled a popup that was never opened, so when the
browser blocked it the account dialog stayed busy indefinitely. It now
rejects straight away and the dialog says the sign-in didn't finish.

* fix(dashboard): password show/hide reads as one action, not a pressed toggle

The button flipped its label between Show and Hide password and also set
aria-pressed, so assistive tech announced "Hide password, pressed" while
the password was showing. It keeps the action label and drops aria-pressed.

* fix(dashboard): screen readers hear a form dialog's success message

The done state arrives as a new status region that already holds its text,
which screen readers don't announce, so after a change succeeded they only
heard "Done, button". The Done button that takes focus is now described by
the message.

* fix(dashboard): requesting an email change no longer sets the cached email

The server only stages the new address until the emailed link is
followed, but the dialog wrote it into window.user.email straight away.
That cached a wrong address for the session and kept refresh_user_data from
updating the Account tab once the change was confirmed, since the two
already matched.

* fix(dashboard): form dialog labels drop the global label shadow

style.css's bare `label` rule gave the field labels a white text-shadow,
which ghosted them on the dark theme, and an extra 3px under each label.

* fix(dashboard): dialog inputs keep a 1px border when focused

style.css's global input :focus rule still set a 2px border width, so a
dialog field's text shifted by a pixel each time it took focus. The focus
ring is the box-shadow; the border stays 1px.

* fix(dashboard): phone dialog inputs stay 16px instead of jumping on focus

On phones style.css's .device-phone input rule outranked the dialog's
16px size at rest, so fields drew at 17px and shrank to 16px the moment
they took focus.

* fix(dashboard): temporary accounts can delete themselves again

The server reports oidc_only for every passwordless account, temporary
ones included, so the shared user-protected helper sent a temporary
account into the linked-provider sign-in, which has no URL for it, and the
deletion never went out. Temporary accounts post directly, as the route
admits them without credentials.
2026-10-22 17:13:50 -07:00
Daniel Salazar 80a9cd85ee fix(ai): chat requests run to their upstream usage report after the caller disconnects (#4302)
- Providers send upstream calls with a driver-owned `upstreamSignal`
  instead of the caller's `abortSignal`. A disconnect detaches the chat
  stream: output stops, the provider keeps reading until the upstream
  reports usage, and that usage is metered.
- A detached stream gets SDK_TIMEOUT_MS to finish. After that the driver
  aborts the upstream, and the stream is metered like any other stream
  that ends without usage.
- The disconnect check also runs right before dispatch.
- The OpenAI stream handlers report usage that arrived before an abort.
- Docs: stopping a stream stops the output, and a request that reached
  the model is billed in full. Detached streams get 10 more minutes.
2026-10-22 16:08:05 -07:00
Nariman Jelveh 63370fcfa5 fix(dashboard): mobile and tablet review — layout, touch, safe areas (#4303)
* fix(dashboard): give narrow Files listings the two-row layout

The two-row rows (name, then size and date) were keyed to a phone-width
viewport, so between 481px and ~930px, where the sidebar and folder column
take their share, the fixed size and modified columns squeezed names to a
few characters. The layout now switches on the listing's own width with a
container query, which phones still meet and wide listings never do.

* fix(dashboard): wrap the Files toolbar onto two rows in narrow listings

On tablets the nav buttons, crumbs and actions sat on one row wider than
the listing, so the sort, view, new-folder and upload buttons ran off the
right edge. Narrow listings now use the phone's two-row toolbar (crumbs
above, buttons below); only the menu-button clearance stays phone-only.

* fix(dashboard): keep a deep folder path on one row with the current folder in view

A long path wrapped the crumbs over several lines on top of the column
headers and pushed the toolbar actions off-screen; where the crumbs already
scrolled (phones), the row opened at the root, hiding the current folder.
The crumbs now scroll within their row and are scrolled to their end on
render and when the tab is shown.

* fix(dashboard): end clipped file names in the list with an ellipsis

With the name column auto-sized, which is always the case on phones and
tablets, a long name was cut off mid-letter with no sign it continued.

* fix(dashboard): keep the Files selection bar under the drawer and sheets

The bar's visible state set z-index 99999999999999999 (clamped to the
maximum), so in phone select mode it drew over the open navigation drawer,
covering the account button, and over the share sheet it opens. Its base
z-index already clears the rows and the footer.

* fix(dashboard): let the last Files rows scroll clear of the phone selection bar

In phone select mode the list's bottom padding only cleared the footer, so
the last row stayed behind the floating action bar even fully scrolled.

* fix(dashboard): keep the phone Files footer and selection bar above the home indicator

The installed app runs edge to edge, so the footer's item count sat under
the iPhone home indicator. The footer now grows by the bottom safe-area
inset, and the list padding and floating action bar move up with it, as
the Apps tab and the dashboard's bottom sheets already do.

* fix(dashboard): clear the home indicator in the item properties sheet

On phones the sheet sits flush with the bottom edge, and unlike the share
sheet it mirrors, its body didn't pad for the bottom safe-area inset, so
the last details scrolled under the iPhone home indicator.

* fix(dashboard): keep the app feedback sheet's buttons above the home indicator

As a phone bottom sheet it sits flush with the bottom edge, so Cancel and
Send ended under the iPhone home indicator.

* fix(dashboard): size the touch context menu as a card on tablets

The touch menu took 90% of the viewport, which on an iPad meant a ~920px
strip with its labels far from the row they belong to. It now stops at
400px, which every phone stays under. Its height is also measured after
the width is set, so the clamp to the viewport uses the real height.

* fix(dashboard): let the Apps modals scroll in a short viewport

The Add-an-app and uninstall cards had no height cap and their overlay
centers without scrolling, so on a landscape phone with the keyboard up
(the viewport resizes for it) the request form's title and buttons sat
off-screen with no way to reach them.

* fix(dashboard): align Account and Security with the other tabs on phones

Below 768px their wrapper added a 16px gutter inside the content area's
own, so both tabs sat 32px in while Home, Apps and Usage (and the menu
button above them) sit at 16px. The extra padding stays for Apps, whose
content area has none.

* fix(dashboard): keep Home's recent apps inside their card on tablets

Each recent app was a fixed 200px inside 1fr columns, whose minimum is
that width, so in the half-width card on a landscape iPad the second
column ran past the card's edge and its name was cut off with the
ellipsis out of view. The rows now shrink with their column.

* fix(dashboard): keep every Files folder reachable in a short viewport

Trash was fixed to the bottom of the viewport, so in a landscape phone or
a short window it sat on top of Public and Videos, while Shared ran below
the column with no way to scroll to it. The folder column now scrolls,
and Trash sits at its foot in the flow: in the same spot when there is
room, after Shared when there isn't.

* fix(dashboard): don't leave Account and Security buttons highlighted after a tap

Their hover background wasn't limited to hover-capable pointers like the
rest of the dashboard's hover styles, so on touch devices a tapped button
(Change Password, Manage Sessions, ...) stayed shaded after its dialog
closed.

* fix(dashboard): give the Files row menu button a full-size touch target

On phones the row's '⋯' is 40x23, right beside a row whose tap opens the
item, so a slightly-off tap opened the file or folder instead of its
menu. In the two-row layout it now has a 44px square hit area on touch
screens.

* fix(dashboard): enlarge the Files back, forward and up touch targets

They were 28px squares on touch screens. On coarse pointers they now take
36px with a matching negative margin, so the toolbar layout is unchanged.

* fix(dashboard): size Home's recent apps card to its apps on phones

Its 310px minimum height matches the square welcome card beside it on
wider screens. In the single-column layout the welcome card drops its
square shape but the minimum stayed, leaving a block of empty card under
a short list of recent apps.

* fix(dashboard): stop clipping app names' descenders on phones and small tablets

Below 768px the tile height budgeted 52px icon + 10px margin + 12px for a
label line that is 15.6px tall, so the bottoms of g, p and y were cut off
in every app name. The tile now has the full line (78px) and the row gap
gives back the 4px, so tiles sit exactly where they did.

* fix(dashboard): show Usage amounts in full on phones

Every Usage table cell ellipsized, numbers included, so on a phone a
token count like 12,345,678 read as "12,345...". Below 480px the units
and cost columns now keep their full width and the resource name is the
one that truncates. Wider layouts are unchanged.

* fix(dashboard): stack Home's recent apps when their card is narrow

In the two-column Home on a portrait tablet (iPad Air and Pro 11 at
820-834px, or 769px windows) the Apps card is about 200px wide, and two
columns of recent apps cut every name to a letter or two. The list now
switches to one column on the card's own width; 1024px and wider keep two.

* fix(dashboard): fit two grid tiles a row in narrow Files listings

Grid view's 170px column minimum left one 330px tile per row on 360px
phones (and in a tablet's narrow listing), so barely two files fit on
screen. Narrow listings now use two equal columns, and the tile's icon
box shrinks with the tile instead of keeping its fixed 150px.

* fix(dashboard): align the Shared and Trash rows' menu button on phones

Item rows get position: relative from jQuery UI draggable, but the Shared
and Trash rows above Home's folders aren't draggable, so their '⋯' was
positioned against the whole list and sat 3px off from every other row's.
2026-10-22 15:39:47 -07:00
Daniel Salazar 631b264c1c test(ai): every AI driver holds and meters at the AI cost factor (#4304)
Image, video, TTS, speech-to-text and voice-changer driver tests run one
request at factor 1 and again with every `ai.cost.factor.*` hook set to
2, and check that both the credit hold and the recorded cost double.
Chat and OCR already had this coverage. A driver that hands a provider
the raw metering service instead of its factored view now fails its
test.

`withAiCostFactorHook` in integrationTestUtil keeps the hook in place
for a callback, since providers record their usage unawaited.
2026-10-22 15:31:15 -07:00
Filip KujundžićandClaude Opus 5.5 809c303117 fix(ai): treat a gateway 4xx with no upstream error body as a route failure (#4301)
Infron (a new-api gateway) answers "bad response status code N" with the
upstream's status when the upstream replied without a JSON error, e.g. a
model the upstream no longer serves. That 4xx was classified as the
caller's bad request, so the request skipped fallback, the dead route
stayed in rotation, and the caller got upstream_bad_request. It now counts
as an upstream failure: fallback runs, the route is marked unhealthy, and
an exhausted chain reports upstream_provider_unavailable. A gateway error
that carries the vendor's own message still stays request-level.

Co-authored-by: Claude Opus 5.5 <[email protected]>
2026-10-22 15:14:28 -07:00
VonypetandCursor afad53d3f4 feat(gui): add Filipino and restore the language menu (#4263)
The desktop built the language list and never placed it on the user menu, so a saved locale could not be chosen. Register Filipino and show that list.

Co-authored-by: Cursor <[email protected]>
2026-10-22 15:13:43 -07:00
Daniel Salazar 8b8dd19eae fix(teams): refuse to boot when teams_enabled has nothing registering Teams (#4300)
* fix(teams): refuse to boot when teams_enabled has nothing registering Teams

Core's seat checks (team 2FA, seat password reset, email and username
changes, self-delete, the signup placeholder claim) read the team store and
skip when it is absent. With `teams_enabled` on and no extension registering
the store and service, the server now fails at boot instead of serving
without those checks.

- whoami no longer string-matches the extension proxy's "missing property"
  error: with the flag on, the store is always there.
- Tests that turned the flag on without Teams drop it; the puter.js harness
  sets it only under the `teams` capability.
- Drop a stale services/index.ts comment.

* refactor(gui): drop the Teams console styles, which ship with the console now
2026-10-22 15:11:36 -07:00
Juan Fernando Castro 1fdde02f6a feat(metering): free and no-allowance plan traits, provisional resolver answers (#4284)
* feat(metering): let registered plans declare free and no-allowance traits

A policy passed to registerPolicy can now set `free: true`, so
isFreeSubscription and everything that reads it count the plan as free.
It can also set `noAllowance: true`: the plan grants no monthly
allowance, so only purchased credit can be spent, and the cached
credit check refuses an actor that has none. A non-positive allowance
without the trait still means the plan is not metered.

getActorSubscriptionStatus returns the plan along with whether it was
paid for and whether the answer is provisional (a lookup behind it
failed, or the actor lacks a field resolvers key on), so callers can
tell a fallback from a resolved plan. getActorSubscription is unchanged.

* feat(metering): provisional resolver answers, and tighter no-allowance accounting

A subscription resolver can answer { id, provisional: true }. The answer
wins like any id but is cached only for the fallback window.

On a no-allowance plan, the month record now records allowanceUsed = 0,
so spend that purchased credit paid for is not counted again against an
allowance the actor gets later in the month. An actor that never had
budget raises no credit-state event. setMonthAllowance refuses plans
whose allowance an override never applies to.

* fix(metering): keep resolved plans clean when a default fails, and announce moves onto no budget

A failed or provisional default resolver now marks the answer provisional
only when the default is what applied; an explicit plan that won is
cached for the full window. A provisional answer with no id still marks
the result provisional.

A plan or credit change now re-arms an actor's credit alert instead of
forgetting it, so an actor moving onto a plan with no budget hears
'exhausted' once. An actor never seen with budget still hears nothing.

* chore(metering): let a provisional answer carry no id, and fix two resolver docs

* chore: one-line comments
2026-10-22 15:47:01 -04:00
Juan Fernando Castro 1627650abc refactor: make teams an optional feature (#4248)
* refactor: let core run without a teams registration

Teams becomes optional: `services.team` and `stores.team` may be absent, and
core stops assuming either is there.

Seventeen call sites across share, signup, account deletion and the homepage
now degrade instead of throwing: a team recipient that cannot be resolved is
"no such recipient" rather than a fault, a group share folds in nothing, and
account deletion stops asking what a seat was charged for. This generalises
the pattern `gates.ts` and `SocketService` already used.

Teams' own files take a private handle to their store and service, so the
feature's code stays free of optionality that only core needs.

No behaviour changes where Teams is registered.

* refactor: put a contract between core and teams

`core/teams.ts` holds the shapes core reads and the calls it makes, and the
registries declare the pair as provided rather than built in. The service,
store and controller are registered by whatever runs the feature.

Typing them properly turned up eight AuthController call sites the previous
commit could not see: the controller reads its stores from the built-in
registry, so an absent `team` fell to an index signature and typed as
`unknown` instead of erroring. They now go through the same optional path as
the rest.

The feature's own tests move with it. The cases in core suites that build a
team to exercise a core rule still need relocating, so those suites are red
here.

* test: stub core's seat rules, rehome the rest with the feature

A core suite can no longer build a team, so the rules that refuse a seat get
`testFixtures/seatStub.ts` instead: it answers the contract, which is all
those rules read, and puts the registry back when the test ends.

Seventeen cases move that way. Four could not: group shares and a seat's 2FA
enrolment over HTTP need the real feature and the `/teams` route, so they go
with it, as does the whole of `GroupGrants.test.ts`.

Two whoami cases asserted a seat's team through `extension.import`, which
resolves the registry rather than the stores object. They now assert the
absent case, and the behaviour they covered is asserted against
`buildUserDetails`, which takes its stores as an argument.

Suite: 12252 passed, 38 skipped.

* refactor: take the teams console out of the built-in tabs

`TabTeams`, `teamsConsole` and `teamActionIcons` are no longer built in. The
dashboard already invites a tab through its `dashboard-will-open` event, so
whatever provides Teams contributes the console the same way.

Its 113 strings go with it and register into the shipped English dictionary
at load, so `i18n('teams_*')` resolves as it did. Only `en.js` ever carried
them, so no translation is lost.

Three modules stay, because things other than the console use them:
`teamBadge` (the dashboard shell), `orgSeat` (Home, Account, Security) and
`shareTeams` (the share picker). They already render nothing when the
account has no team. `teams_account_of` stays with `teamBadge` for the same
reason, and the `share_*` and `plan_team_*` strings stay with their callers.

The GUI builds clean with no console at all, and builds it in when the tab
is contributed.

* docs: seats past the owner's current limit get no allowance
2026-10-22 15:32:39 -04:00
Daniel Salazar 4968a69d45 fix(s3): keep credential-chain credentials an hour from expiry when signing (#4299)
Presigned URLs die with the credentials that signed them, and the SDK only refreshes temporary credentials five minutes before expiry, so a URL signed late in a credential's life could expire long before its requested lifetime (up to an hour). The chain provider now reports expiry an hour early and force-refreshes credentials that are already inside that window.
2026-10-22 00:47:48 -07:00
Juan Fernando Castro aa47dcd3e5 feat(workers): internal teardown that takes the script down before the row, and destroy reports Cloudflare refusals (PUT-2219) (#4297) 2026-10-21 18:50:54 -07:00
Daniel Salazar c8424f6775 fix(fetch): raise HTTP/2 receive windows for plain fetch (#4296)
Node 26's bundled undici negotiates HTTP/2 by default with a 256 KB
per-stream receive window, which capped large downloads at about
window / RTT (8.6 MB/s vs 46 MB/s over HTTP/1.1 at 23 ms). The guard swaps
in an Agent of the same class with 4 MB stream / 16 MB connection windows.
2026-10-21 13:54:31 -07:00
Daniel Salazar b3c85c86e9 feat(puter.js): open the upgrade page with what was refused (#4294)
The web upgrade dialog's button opens the dashboard with the refusal
reason, the app the call ran as, and the refused method as query params,
so the dashboard can open its upgrade flow with that context.
2026-10-21 13:09:02 -07:00
Daniel Salazar d22d419c4d chore: move to Node 26 (#4295)
* chore: move to Node 26

Docker image and CI run on Node 26 (bookworm-slim, same Debian base as before).

Node 26's bundled undici reads the global dispatcher from the
`undici.globalDispatcher.2` slot, so the guard's 10-minute timeout wrap on
`.1` no longer reached plain fetch() and provider calls fell back to the
5-minute headers timeout. The guard now wraps every present slot and tracks
wrapped dispatchers in a set, since compose() can return a Proxy that writes
a marker property through to the original dispatcher.

* test(puter-js): disable Node's web storage in test workers

Node 26 defines its own localStorage global, undefined without a storage
file, and vitest's jsdom environment doesn't replace globals that already
exist, so jsdom tests saw localStorage as undefined.
2026-10-21 13:04:23 -07:00
Daniel Salazar 0760530d0c feat(gui): edit display name and bio from the Account tab (#4293) 2026-10-21 12:57:15 -07:00
Juan Fernando CastroandDaniel Salazar e18a5c2b54 fix(apps): refuse claimed subdomains as app index_url hosts (PUT-2110) (#4267)
index_url may not point at a system host: the origin, the api host, or a
subdomain an extension has claimed. App create and update now apply the
same refusal to `<claimed>.<domain>` (any case, any port) that they apply
to the origin and api hosts.

The host-to-claim match is a new exported `subdomainLabelOf` in
hostRedirects, also used by the claim dispatch middleware, so both
decide the same way.

Co-authored-by: Daniel Salazar <[email protected]>
2026-10-21 13:19:31 -04:00
Kailasnath TandDaniel Salazar 75385501a7 fix(fs): support offset and byte_count in read operations (#4266)
* fix(fs): support offset and byte_count in read operations

Allow partial content reads via offset and byte_count parameters in puter.fs.read and backend FS controllers.

- In puter.js read operation, check options.offset and options.byte_count against null/undefined so offset: 0 is preserved in query params.
- In handle_resp, accept any 2xx status code so 206 Partial Content responses resolve successfully.
- In backend FS controllers (LegacyFSController and FSController), derive the HTTP byte range from query offset/byte_count parameters when an explicit Range header is not present.
- Add unit and integration tests across controllers and SDK test suites.

* fix(fs): strict range params, 416 past EOF, drop format churn

- resolveReadRange accepts only decimal digit strings and safe integers
- readContent maps S3 InvalidRange to 416 instead of a 500
- revert unrelated union-type reformatting, restore truncated test comment

---------

Co-authored-by: Daniel Salazar <[email protected]>
2026-10-20 21:20:51 -07:00
Daniel Salazar 5b0205f2bf fix(hosting): apps publish with write again (#4291)
Reverts the hosting part of #4290. manage stays a user-to-user grant;
an app with write on its user's directory can publish and keep serving it.
2026-10-20 17:36:36 -07:00
Daniel Salazar 3bce3b056e misc: hardening (#4290) 2026-10-20 17:12:13 -07:00
dependabot[bot] f70240784f chore(deps): bump follow-redirects from 1.16.0 to 1.16.1 (#4287)
Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects) from 1.16.0 to 1.16.1.
- [Release notes](https://github.com/follow-redirects/follow-redirects/releases)
- [Commits](https://github.com/follow-redirects/follow-redirects/compare/v1.16.0...v1.16.1)

---
updated-dependencies:
- dependency-name: follow-redirects
  dependency-version: 1.16.1
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-20 16:53:12 -07:00
dependabot[bot] 356e7b2b94 chore(deps): bump follow-redirects in /src/puter-js (#4288)
Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects) from 1.16.0 to 1.16.1.
- [Release notes](https://github.com/follow-redirects/follow-redirects/releases)
- [Commits](https://github.com/follow-redirects/follow-redirects/compare/v1.16.0...v1.16.1)

---
updated-dependencies:
- dependency-name: follow-redirects
  dependency-version: 1.16.1
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-20 16:29:27 -07:00
jelveh f02b4a6b73 feat: tell apps when a launched item is a folder; Open with AI above Open With
A folder opened with an app (Open With, Open with AI) reached puter.js as
a file: the GUI never forwarded the signature's fsentry_is_dir, so the
launched FSItem always had isDirectory false. The GUI now appends
puter.item.is_dir, and puter.js reads it in onLaunchedWithItems and the
onItemsOpened launch replay. Both now share one URL-to-FSItem builder.

"Open with AI" moves directly under Open, above Open With, in the
desktop and dashboard file menus.
2026-10-20 16:16:20 -07:00
dependabot[bot] 0768978111 chore(deps-dev): bump vitest from 4.1.11 to 4.1.12 (#4275)
Bumps [vitest](https://github.com/vitest-dev/vitest/tree/HEAD/packages/vitest) from 4.1.11 to 4.1.12.
- [Release notes](https://github.com/vitest-dev/vitest/releases)
- [Changelog](https://github.com/vitest-dev/vitest/blob/main/docs/releases.md)
- [Commits](https://github.com/vitest-dev/vitest/commits/v4.1.12/packages/vitest)

---
updated-dependencies:
- dependency-name: vitest
  dependency-version: 4.1.12
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-20 15:47:43 -07:00
Nariman Jelveh 66093e78f3 feat(gui): "Open with AI" in the dashboard Files menu, opening like Open With (#4289)
"Open with AI" now launches the `ai` app with the item the same way
Open With launches any app: launch_app with file_path/file_uid, so the
app gets a signed launch item via puter.ui.onLaunchedWithItems. It
replaces the custom `ai:openFsEntries` postMessage, which went to an
existing AI window.

- The helper moved from UIItem into helpers/openWithAI.js, shared by the
  desktop item menu and the dashboard Files menu (generate_file_context_menu).
- A shortcut opens its target.
- Dropped from multi-select menus, since a launch opens a single item.
- Renamed open_in_ai -> open_with_ai ("Open with AI") and removed the
  unused ai_app_unavailable string.
2026-10-20 15:33:12 -07:00
dependabot[bot] ed531e3992 chore(deps): bump undici from 7.30.0 to 7.30.1 (#4280)
Bumps [undici](https://github.com/nodejs/undici) from 7.30.0 to 7.30.1.
- [Release notes](https://github.com/nodejs/undici/releases)
- [Commits](https://github.com/nodejs/undici/compare/v7.30.0...v7.30.1)

---
updated-dependencies:
- dependency-name: undici
  dependency-version: 7.30.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-20 15:27:17 -07:00
dependabot[bot] 53e27f3d41 chore(deps): bump mysql2 from 3.24.5 to 3.24.6 (#4279)
Bumps [mysql2](https://github.com/sidorares/node-mysql2) from 3.24.5 to 3.24.6.
- [Release notes](https://github.com/sidorares/node-mysql2/releases)
- [Changelog](https://github.com/sidorares/node-mysql2/blob/master/Changelog.md)
- [Commits](https://github.com/sidorares/node-mysql2/compare/v3.24.5...v3.24.6)

---
updated-dependencies:
- dependency-name: mysql2
  dependency-version: 3.24.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-20 15:25:23 -07:00
dependabot[bot] deac121675 chore(deps-dev): bump eslint-plugin-prettier from 5.5.6 to 5.5.7 (#4278)
Bumps [eslint-plugin-prettier](https://github.com/prettier/eslint-plugin-prettier) from 5.5.6 to 5.5.7.
- [Release notes](https://github.com/prettier/eslint-plugin-prettier/releases)
- [Changelog](https://github.com/prettier/eslint-plugin-prettier/blob/main/CHANGELOG.md)
- [Commits](https://github.com/prettier/eslint-plugin-prettier/compare/v5.5.6...v5.5.7)

---
updated-dependencies:
- dependency-name: eslint-plugin-prettier
  dependency-version: 5.5.7
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-20 15:22:09 -07:00
Nariman Jelveh 53e6e3179b refactor(gui): remove the AI side panel and its desktop button (#4286)
The desktop no longer launches the `ai` app at boot. This removes the
`is_panel` window mode, which existed only for that launch, and the
toolbar/taskbar/window/popover offsets tied to it. It also removes the
desktop sparkles button that showed the panel, and the
`silent_on_failure` launch option, whose only caller was the boot launch.
"Open in AI" in the item context menu still launches the app as a
normal window.
2026-10-20 15:21:21 -07:00
dependabot[bot] 73624bf1d0 chore(deps): bump smtp-server from 3.19.21 to 3.19.23 (#4276)
Bumps [smtp-server](https://github.com/nodemailer/smtp-server) from 3.19.21 to 3.19.23.
- [Release notes](https://github.com/nodemailer/smtp-server/releases)
- [Changelog](https://github.com/nodemailer/smtp-server/blob/master/CHANGELOG.md)
- [Commits](https://github.com/nodemailer/smtp-server/compare/v3.19.21...v3.19.23)

---
updated-dependencies:
- dependency-name: smtp-server
  dependency-version: 3.19.23
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-20 15:19:00 -07:00
Nariman Jelveh 1a9e43f5e5 fix(dashboard): adversarial review — bugs, a11y, perf, i18n (#4285)
* fix(dashboard): size the Notifications sidebar row like its neighbors

The bell button's font: inherit overrode the sidebar item's font-size
(16px instead of 14px, 15px in the drawer), and its icon wrapper stayed
18px while the drawer grows nav icons to 20px.

* test(dashboard): scope the directory-note assertion to the directory row

The 2FA row reuses .teams-directory-note, so the unscoped selector read
both notes and the test failed.

* fix(dashboard): only treat a picked file as a new avatar while the avatar picker is open

The Account tab listened for every file_opened on the dashboard window,
so choosing an image for a web link's Change Icon in the Files tab opened
the profile-picture crop modal, and saving it replaced the avatar.

* fix(dashboard): let dialogs over the Files tab keep their keyboard

The Files keydown handler ran under an open confirm, so pressing Enter on
the focused Delete button of the permanent-delete prompt opened the
trashed item instead (stacking a second alert), and arrows or Delete
acted on the rows behind the dialog.

* fix(dashboard): keep Apps grid keys away from an open context menu

Choosing a tile's menu item with Enter (e.g. Open in new tab) also
launched the focused app in place, and arrows moved tile focus while
they walked the menu.

* fix(dashboard): Escape on a menu or modal inside an app folder keeps the folder open

The menu's and the uninstall modal's Escape handlers removed them before
the folder's listener ran, so it saw nothing above it and closed the
folder with the same keystroke.

* fix(dashboard): a tap on the drawer or sheet scrim only dismisses it

Outside-press handlers closed the mobile drawer and the notifications
sheet on touchstart/pointerdown, which took away the scrim's hit area
before the click, so the tap also landed on the app tile or file row
underneath.

* fix(dashboard): let the sidebar nav scroll on short viewports

The nav clipped its overflow, so in a landscape phone's drawer (or a
short desktop window) Usage, Account, and Security fell below the fold
with no way to reach them.

* fix(dashboard): keep the mobile drawer full width when the desktop sidebar is collapsed

The remembered collapse outranked the drawer's width rule, so below the
drawer breakpoint it opened as an unlabeled 50px icon strip, and the
toggle that would undo it is hidden there.

* perf(dashboard): stop refetching account data each time focus leaves an app

Home's window focus listener fired whenever focus came back from an app's
iframe, so every click from an app to the dashboard re-pulled whoami,
subscription, storage, and usage (5 requests), on any tab. It now only
refreshes after the page itself lost focus to another window.

* fix(dashboard): show a seat's team name unescaped in the plan card

i18n escapes its arguments for HTML, but the note is set with .text(), so
a team called "Bob's & Co" read "Managed by Bob&apos;s &amp; Co".

* fix(dashboard): stop double-escaping app names in uninstall alerts

UIAlert escapes its message, so an app titled "Tom & Jerry" read
"Tom &amp; Jerry" in the uninstall-failed and sessions-not-ended alerts.

* fix(dashboard): roll back an uninstall whose revoke failed

puter.perms.revokeApp resolves with error set instead of rejecting, so a
failed revoke still removed the tile, marked the app removed for later
sessions, and ended its sign-in, while the app kept its permissions and
the rollback and error alert never ran.

* fix(dashboard): read a team account's suspension when its name matches an Object member

The per-account state map was a plain object, so a username such as
constructor or toString was already "in" it, its audit entries were
skipped, and a suspended account showed as active with no Delete.

* fix(dashboard): update the profile card and sidebar name after a username change

Neither carried the .username class that the rename refreshes, so both
kept showing the old name until reload.

* fix(dashboard): ask how to restore a trashed item whose name is taken

Restore moved without overwrite or dedupe, so when the original location
already had an item of that name the move 409'd and nothing happened:
an uncaught error from the item menu, a console line from the others.
All three restore entry points now share one path that offers the
Replace / Keep Both prompt the paste flow uses, alerts on other errors,
and updates the footer and empty notice once the rows are gone.

* fix(dashboard): refresh a renamed row's name, icon, and position in the Files tab

The Files tab passed rename_file a callback it never accepted, so a
renamed row kept the desktop's 20-character truncation even in a wide
column, kept its old type icon, and stayed out of sort order. rename_file
now takes an optional onDone (carried through undo), which the Files tab
uses to fit the name to its column, refresh the icon and type, and move
the row into place. A cancelled rename no longer truncates the name
either.

* fix(dashboard): drop Publish and Properties from the Shared view's menu

The Shared view is a query, not a folder, so both acted on the virtual
shared path: Properties showed an unknown error and Publish opened a
dialog for a path that doesn't exist.

* fix(dashboard): close the Files image preview on Escape and when leaving the folder

Escape cleared the selection but left the Space preview open with
nothing selected, the preview stayed over a different folder after
navigating, and a preview whose URL arrived after it was dismissed or
replaced still opened.

* fix(dashboard): don't move Files history for a navigation that's dropped

renderDirectory ignores a navigation while another folder is loading,
but every caller had already pushed or stepped history, so after a click
during a load the listing and history disagreed and the next Back or
Forward seemed to do nothing. History now only moves when the render
will run; back/forward steps share one helper.

* fix(dashboard): reopening the current folder adds no Files history entry

Clicking the folder already on screen (e.g. Home in the sidebar) pushed
a duplicate entry, so the next Back reloaded the same folder.

* fix(dashboard): show installed apps when the recommended list fails to load

get-launch-apps sat in the same Promise.all as the installed apps and
its body was parsed without a status check, so a network error or a
proxy's HTML error page replaced the whole grid with "Failed to load
apps". A failed list now falls back to the last one that loaded.

* fix(dashboard): fit the app uninstall dialog on narrow phones

The card is mounted outside the dashboard's border-box rule, so its
320px min-width plus padding came to 370px and was clipped on 360px
screens; a long unbroken app title also overflowed it. Desktop sizing is
unchanged.

* fix(dashboard): keep a new team account's one-time password on screen until dismissed

The credential box lived only in the DOM inside the panel that every
refresh repaints, so revisiting the Teams tab or any later action wiped
a password that can't be retrieved again. It is now kept in state, for
the team it belongs to, and drawn with the panel.

* fix(dashboard): reload the Teams tab after a change made while it was loading

refresh() handed back any load already in flight, so an action or team
switch made during a revisit's background load awaited data read before
the change: a toggled setting snapped back, and a team switch could
paint one team's roster next to another's record. A change during a load
now queues one more load, the stale one skips its paint, and revisiting
the tab doesn't start a second load while init's is still running.

* fix(dashboard): send one re-enable request for a double click in Teams

Re-enable has no confirm dialog to absorb a second click, and nothing
held the action while it ran, so a double click enabled the account
twice (two audit entries). A suspend or re-enable already in flight for
an account now ignores further clicks.

* fix(dashboard): revert the avatar and say so when saving a new photo fails

The crop's Save showed the new picture everywhere and revealed Remove
without waiting for the upload, and update_profile swallows failures,
so a failed save looked saved until reload. The save is now awaited: a
failure puts the previous picture back with an error hint, and Remove
waits for the upload so it can't race it.

* fix(dashboard): hide the Usage tab's plan button for team seats

A seat's plan belongs to its team, so Home already hides its upgrade
link; the Usage tab still offered Upgrade, which only reaches a refusal.

* fix(dashboard): don't drop Home's forced usage refresh during a running load

loadUsageData handed any caller the load already in flight before looking
at force, so a subscription change (or a return from the billing portal)
during a routine load was answered with data read before the change, and
the plan card stayed stale until the next refresh. A forced call now runs
once more after the running load.

* fix(dashboard): activate Apps when the URL names an unknown tab

An unknown hash rendered Apps as the active tab but skipped its
activation, so the grid kept the generic content padding and Apps'
onActivate never ran.

* fix(dashboard): make the sidebar user menu reachable from the keyboard

The user button was a div with no tabindex, so Log Out, account
switching, and Contact Us were out of reach without a pointer. It is
now a button (rendering unchanged), with aria-haspopup.

* fix(dashboard): name the sidebar's collapse and drawer buttons for assistive tech

The collapse toggle held only two unlabeled images, so screen readers
announced an unnamed button; it now carries a translated label and
aria-expanded that follow its state. The drawer's open/close labels were
hardcoded English and now go through i18n.

* fix(dashboard): make Home's recent apps reachable from the keyboard

Each recent app was a click-only div, so keyboard users couldn't open
one. They are now buttons with a focus ring; the icon is decorative
since the title names the button.

* fix(dashboard): name the Security tab's 2FA switch for assistive tech

The checkbox sat in a text-less label, so screen readers announced an
unnamed checkbox. It now carries the translated Two Factor
Authentication label, as the Teams switches do.

* fix(dashboard): sort the Usage table from the keyboard

The sortable headers were click-only th cells. Each now holds a button,
carries aria-sort, and keeps focus across the re-render a sort causes.
The Resource/Units/Cost headers go through i18n.

* fix(dashboard): don't rebuild the Files listing under an open menu

Returning to the browser tab re-rendered the folder even with a context
menu or action sheet open, so its actions then targeted detached rows:
Rename did nothing, Restore faded a row that was already gone. The
refresh now waits while a menu is up, as it does for a rename.

* fix(dashboard): point the Files sidebar's Documents at the current home

The entry read window.documents_path, which globals.js sets once from
the cached user and nothing updates; login and username changes keep
window.docs_path current instead. After a username change, Documents
pointed at the old home and showed "This folder couldn't be opened",
even after reloading.

* fix(dashboard): let only the newest share listing render in the share dialog

Each action ends in a refresh, and refreshes from overlapping actions (or
the opening load) could finish out of order, so an older listing landed
last and undid what the user just did, e.g. the link control snapped
back to restricted after turning the link on. Older refreshes now drop
their result.

* perf(dashboard): load a team's record, roster, and plan in parallel

loadSelected awaited the audit log (every page of it), then the members,
then the plan, on every visit to the Teams tab. They don't depend on
each other, so they are now fetched together.

* fix(dashboard): keep keyboard focus on the Teams control that was used

Every Teams action repaints the whole panel, which dropped focus to
<body>: a keyboard user toggling a switch or suspending an account was
sent back to the top of the page. The repaint now refocuses the same
control, or the same account's new action when the old one is gone.

* fix(dashboard): give the app uninstall prompt dialog semantics and focus

The prompt had no dialog role or label and never took focus, so keyboard
and screen-reader users stayed on the inert grid behind it. It is now a
labelled modal dialog that focuses Cancel on open and returns focus to
the app's tile when dismissed.

* fix(dashboard): keep the add-an-app tile when there are no apps

With no apps at all, the empty state replaced the whole grid, add tile
included, so a new user on an instance with no recommended apps had
nothing to act on. The empty line now sits above a grid that keeps the
tile, as a search with no matches already does.

* fix(dashboard): don't rebuild the Apps grid under an active drag

An uninstall finishing (or a failed one rolling back) re-rendered the
grid even while another tile was being dragged, replacing the dragged
tile with a fresh copy: the copy showed under the ghost and the drop
was lost. Renders asked for mid-drag now wait for the one the drop does;
the folder eject, which rebuilds the grid as part of the drag, still can.

* fix(dashboard): return focus to the drawer toggle when the mobile drawer closes

Closing the drawer (Escape, the close button, a tab pick) left focus on
controls that had just become hidden, so it fell to <body>; closing the
notifications sheet opened from the drawer tried to refocus the bell
inside the closed drawer. Both now land on the drawer toggle.

* fix(dashboard): truncate Files names between graphemes, with a binary search

The middle truncation sliced UTF-16 code units, so at many widths a long
name with emoji showed a half surrogate (rendered as a replacement
character) or a broken joined emoji. It also measured one more character
per step, a measureText call per character of every long name on each
render and column resize. The cut now falls between graphemes and is
found by binary search; output for other names is unchanged.

* fix(dashboard): translate the Home tab's hardcoded English

The tab label, the recent-apps empty state, card subtitles, the plan
badge and trial wording, the upgrade/manage links, and the usage cards'
"Used" / "of" lines were English literals. They now go through
i18n, with placeholders where values are interpolated.

* fix(dashboard): translate the sidebar's remaining English

The Apps, Files, and Usage tab labels, the user button's fallback name,
and the user menu's Developers item were English literals; they now use
i18n keys like the other tabs.

* fix(dashboard): translate the Usage tab's hardcoded English

The subtitle, the "used of" meter lines (now the existing used_of
key), Usage Details, the plan button, Show more/less, and the details
error were English literals.

* fix(dashboard): translate the Account and Security tabs' hardcoded English

Both subtitles, the delete-account description, the sessions hint, the
username fallback, and the avatar picker's window title were English
literals.

* fix(dashboard): translate the Apps tab's hardcoded English

The uninstall prompt and its failure alert, empty and no-match notices,
pager and folder-dot labels, the search placeholder, the edit toggle,
and the tile menu's Open in New Tab / Quit / Uninstall were English
literals.

* fix(dashboard): translate the Files tab's hardcoded English

The folder sidebar, footer counts (now the existing item-count keys),
sort and view menus, upload status and errors, the empty and
can't-open notices, the reserved-name alerts, the move error, and the
loading text were English literals.

* fix(dashboard): translate the action sheet's Back row and match delete in any language

The touch action sheet styled an item red when its label contained the
English word "delete", so Delete lost its danger styling in every other
language; it now compares against the translated Delete labels. Its
submenu Back row and the properties panel's Directory/disassociate
labels were English literals.

* fix(dashboard): make the Files sidebar, crumbs, and nav arrows keyboard-operable

The folder sidebar, the breadcrumbs, and the back/forward/up arrows were
click-only li/span/img elements, out of the tab order. They now take
focus, carry button/link roles (the arrows with labels and
aria-disabled), answer Enter and Space, and show a focus ring. The rows'
Enter/Space shortcuts also no longer swallow those keys from a focused
control: with rows selected, Enter on Upload opened the selection.

* perf(dashboard): read grid name widths before writing any in the Files tab

fitItemNames read each tile's name width right after writing the
previous name, forcing a layout per row. Reading them all first, then
writing, cuts switching 400 files into grid view from about 300ms to
170ms; the rendered names are identical.
2026-10-20 13:44:42 -07:00
242a654166 feat(ai): add Kling for video, image and speech generation (#4274)
* feat(ai): add Kling for video, image and speech generation

One `kling` API key now registers a provider on three drivers:

- txt2vid: Kling 3.0 Turbo, 3.0, 3.0 Omni, O1, 2.6 and 2.5 Turbo,
  with first/last frames, reference images on the omni models, native
  audio, and editing a source video on 3.0 Omni and O1. Billed per
  second at the resolution, audio and video-input rate; an edit is held
  at the longest source and billed for the clip Kling returns.
- txt2img: Kling Image 3.0, 3.0 Omni, O1 and 2.1, billed per image.
- txt2speech: Kling TTS with its English and Chinese voices, billed per
  request.

Kling's business error codes map onto the driver error contract, so an
exhausted resource pack reports as credit exhaustion and a submit-time
content block as `moderation_flagged`. "Risk control" failures now
count as content-filter refusals.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(ai): treat Kling's unknown-model codes as a Puter-side failure

Kling answers 1202/1203 when the method or model isn't one it serves.
Puter picks both, so these meant catalog drift but surfaced as a 400 that
blamed the caller and raised no alarm. They now return 502 upstream_failed
with a warning alarm.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(ai): tighten Kling input checks and error mapping

- 3.0 Turbo takes at most 2,500 prompt characters with a first frame,
  the image-to-video endpoint's limit.
- O1 refuses reference images together with a first and last frame.
- 3.0 Omni accepts a source of up to 15.5 seconds, and an edit is held
  at 16 seconds to cover it.
- A source video Kling can't decode fails as 400 upstream_bad_request
  instead of a Puter-side 502.
- A 1201 about the model counts as catalog drift, like 1202/1203.
- Auth failures no longer pass Kling's console wording to the caller.

Adds integration tests for the image, video and TTS providers, gated
on PUTER_TEST_AI_KLING_API_KEY.

* feat(ai): wait up to 14 minutes for a video job

Video jobs, Kling edits in particular, can take over 10 minutes. The
poll window and the Gemini Omni deadline move to 14 minutes, under the
SDK's 15-minute idle timeout, with holds and signed source links
lengthened to match. The keep-alive default moves to 920 seconds so it
stays above a 15-minute proxy idle timeout.

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
Co-authored-by: Daniel Salazar <[email protected]>
2026-10-20 12:32:19 -07:00
Daniel Salazar 41e4b582d3 fix(subdomains): emit subdomain.delete for sites deleted with an app (#4273)
Sites and workers tied to an app go with it by FK cascade, on app delete and
on deleting the app owner's account. Only caches were dropped for them, so
the subdomain.delete listeners (CDN purge, worker script teardown, worker
sessions, site cleanup) never ran.
2026-10-20 12:31:40 -07:00
Reynaldi Chernando 2a49689fad fix contradicting jsdocs and docs (#4282) 2026-10-20 18:39:49 +07:00
Juan Fernando CastroandDaniel Salazar b54b76715a feat(fs): finish a home rename that stopped partway (#4255)
* feat(fs): finish a home rename that stopped partway

FSEntryStore.rebaseUserSubtree runs renameUserHome's owner-scoped
cascade on its own (same subtree matching, same rebased-cache
invalidation) and also drops the home row's cached copies at either
path. countUserSubtree counts the owner's rows under a prefix with that
same matching, from the primary. renameUserHome now uses the shared
cascade.

* fix(telemetry): resolve the tracer per span so a test's provider is the one used

withSpan held a ProxyTracer built at module load, which caches the first
provider it delegates to. Test files sharing a worker then sent spans to a
provider an earlier file had already shut down.

* fix(fs): drop cached copies of only the rows a home rebase moves

The owner-scoped rewrite read its invalidation set from everything under the new prefix, which for a finishing rebase is the whole live home; it now reads the rows it is about to move.

---------

Co-authored-by: Daniel Salazar <[email protected]>
2026-10-20 00:59:40 -07:00
Juan Fernando Castro d978538c9d feat: let a subdomain claim keep the main pipeline (#4268)
`extension.claimSubdomain(name, { mainPipeline: true })` serves
`<name>.<domain>` the way core's own subdomains are served: through CORS,
body parsing, auth and every route, `subdomain: '*'` ones included. Routes
declared for the name are ordinary routes, with no option restrictions.
Gated claims (`{ authorize, checkRoute? }`) behave as before.

onlyoffice's subdomain now comes from its extension's claim, so core no
longer lists it among the subdomains it serves itself. Without that claim
`onlyoffice.<domain>` gets the unclaimed-subdomain 404 like any other name.
2026-10-20 00:43:54 -07:00
e83e27f306 feat(ai): edit a source video with txt2vid (Gemini Omni, FLUX 3) (#4272)
* feat(ai): edit a source video with txt2vid on Gemini Omni

txt2vid takes an `inputVideo` option (sent as `input_video`): a URL, data
URI or Puter file path to an MP4, QuickTime or WebM video of up to 32 MiB.
Gemini Omni receives it inline next to the prompt, which describes the
edit or extension. Google enforces the 10-second limit.

Models declare `videoInput` in the catalog, and the driver rejects a
source video for any other model instead of letting the provider drop it.
The credit hold adds 16,000 input tokens for a source video, since
countTokens reports far fewer tokens than the reply bills.

loadFileInput takes an optional `timeoutMs` for web URLs, and cancels
the body of a failed response.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* feat(ai): edit a source video with FLUX 3 on Together AI

FLUX 3 takes `input_video` as Together's `media.source_video`. A web URL
goes through for Together to fetch; a data URI or Puter file is read with
the caller's access, capped at 32 MiB, sniffed as a video and sent inline.
Editing is estimated and billed at the 1080p rate of $0.54/s, since FLUX 3
jobs report no cost. Image inputs can't be combined with a source video.

Seedance 2.0 ignores `source_video`, treats `reference_videos` as a loose
reference rather than an edit, and Seedance 2.5 fails on reference videos,
so neither declares `videoInput`.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* docs(ai): name FLUX 3 as a video editor in the AI overview

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(ai): match Hailuo 02 to Together's price and document estimate billing

Together quotes $0.56 for a Hailuo 02 clip; the catalog had $0.49.

Together's video jobs no longer report what they cost, so every Together
clip is billed at the catalog's highest published rate. The docs said the
charge was the amount Together reports; they now describe the estimate.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix: enforce video edit durations and propagate source fetch cancellation

* fix(ai): pass media inputs as Puter files and stream source videos

Source videos were read whole into the API process, base64-encoded and
sent twice to Gemini (countTokens and the interaction), which held about
128 MiB per in-flight edit for up to ten minutes.

- Video inputs are a URL, a Puter file path or a data URI of at most
  1 MiB. Together gets a Puter file as a 15-minute presigned S3 URL and
  fetches it itself; Gemini streams it into the request body with an
  exact Content-Length, so it's never held as one string.
- Gemini counts only the prompt and image for a source video, and both
  providers read the source only after the credit hold.
- txt2img and txt2vid image inputs accept Puter file paths, read within
  the caller's access right before the provider call. Existing URL,
  data URI and base64 inputs are unchanged.
- puter.js writes a Blob, File or data URI over 1 MiB to a temporary
  file under .tmp, sends its path, and deletes it once the call settles.
- FLUX 3 edits bill at the 720p editing rate, matching their 1280x704
  output.

* fix(ai): stage inputs after website sign-in, keep images working without a write

- puter.js chose the temporary file path before signing a website visitor
  in, so it fell outside the app's data directory and the upload was
  refused. It now signs in first, then picks the path.
- An image that can't be written (no access, storage full) is sent inline
  as before instead of failing the call. Videos still fail, since a
  large inline video isn't accepted.
- Telling raw JPEG base64 from a path decodes only the first 24
  characters instead of the whole input.

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
Co-authored-by: 404oops <[email protected]>
Co-authored-by: Daniel Salazar <[email protected]>
2026-10-19 23:33:03 -07:00
Udit AgarwalandDaniel Salazar 54f542ace4 fix(gui): refresh Start menu apps without blocking cached rendering (#4244)
* fix(gui): refresh Start menu apps without blocking cached rendering

* docs: remove desktop Start menu limits from SDK reference

* fix(gui): drop the 1 MiB cap on the launch-apps refresh

Recent entries carry each app's raw icon column, which can still hold a data URL of up to 5 MB, so a large icon tripped the cap on every request, including the one at startup, and left the Start menu with no apps. Replace the cap's test with one for the unlaunchable-app filter.

---------

Co-authored-by: Daniel Salazar <[email protected]>
2026-10-19 16:13:55 -07:00
Daniel Salazar 18b96d529c fix: misc scan fixes (#4270) 2026-10-19 12:32:13 -07:00
Kunal Jaiswal f146da16d1 fix(gui): drop deleted apps from the start menu (#4243)
The start menu read a session-cached launch-apps list and only refetched when it was empty, so an app deleted after page load (for example in Dev Center) stayed in Recent and Recommended until a reload. Refetch on every open, and filter out entries the backend marks unlaunchable, so a reclaimed or inaccessible app leaves the list too. Apply the same filter to the dashboard's Recent apps. Adds helpers/launchAppsList.js with filterLaunchableApps and its unit test.
2026-10-19 10:50:36 -07:00
Daniel Salazar dd56d54753 docs(bug-bounty): add eligibility rules (#4269)
Bounties cover user security/privacy and platform security/health only.
Reports must show a direct attack without pre-assumed access, and
exploits that rely on sharer/sharee trust are hardening or lower
severity. Also generalizes the metering non-issue, adds existence-only
oracles, and tightens duplicate, proof and first-contact rules.
2026-10-19 10:49:12 -07:00
Daniel Salazar e5fd30ccf0 fix(ai): validate large base64 by decoded length, not regexps (#4265)
* fix(ai): validate large base64 without anchored regexp loops

Gemini TTS replies over ~4 MB of base64 (~65 s of audio) failed with
"Maximum call stack size exceeded". Once a process holds enough compiled
code, V8 compiles new regexps unoptimized, and `^[...]+={0,2}$` then
pushes a backtrack entry per character until its stack overflows.

TTS now searches for a character outside the alphabet instead. The raw
base64 icon/reference-image check drops its own anchored regexp;
decodeStrictBase64 already rejects the same inputs.

* perf(ai): validate base64 by decoded length instead of regexp passes

Unoptimized regexp scans dominated large payloads: 186 ms for max-length
TTS audio, 880 ms in decodeStrictBase64 for 30 MB of video. Buffer.from
skips characters outside the alphabet, so comparing the decoded length
with the one the input length predicts catches them for free.

decodeStrictBase64 now does that plus rejecting the URL-safe alphabet
(~10 ms for 30 MB), and TTS reuses it instead of its own check.
Non-canonical trailing bits now decode instead of being rejected.

* fix(ai): reject non-ASCII base64 that Node decodes by its low byte

Node reads code units above 0xFF by their low byte, so U+012B decodes
as '+' and keeps the decoded length the input predicts. Require ASCII
(~1-2 ms for 43M chars) before the length check.
2026-10-18 21:56:29 -07:00
Neal Shah 517efda38a automerge emergency fix (#4264) 2026-10-19 00:18:58 -04:00
Neal Shah ea3801fb74 PUT-2058 and PUT-2096 (#4256) 2026-10-18 23:30:01 -04:00
Daniel Salazar 0786cc7db2 fix(webdav): PROPPATCH checks write access and answers with the masked path (#4260)
PROPPATCH now requires write access on the target, like PUT and DELETE,
and its response href goes through the request's share-path masker the
way PROPFIND's do.
2026-10-18 00:33:54 -07:00
Daniel Salazar 592020c899 docs(bug-bounty): list re-claimed abandoned site and worker names as a known non-issue (#4261) 2026-10-18 00:33:49 -07:00
Daniel Salazar 9ff224e22f perf(backend): cut sequential round trips on auth, fs, driver, event and site paths (#4259)
Pipeline
- CORS preflights send Access-Control-Max-Age: browsers stop re-preflighting authed calls every 5 s.
- `req.ua` is parsed on first read instead of on every request.
- JSON `rawBody` reuses the parsed buffer instead of copying it.
- `emitAndWait` returns early when nothing listens: no span or closure per unlistened driver call.
- SQLBatcher sends a lone query on the next tick while fewer than 8 flushes are in flight, and
  waits `maxTimeInQueue` as before otherwise. Lone reads skip the 10/30 ms wait; batching under
  load is unchanged.

Auth and permissions
- Token-to-actor auth starts the user, session and app lookups together and checks them in the
  same order: fewer sequential Redis round trips per authed request.
- get-user-app-token and check-app reuse the resolved app row. getOrCreateApp skips the re-select
  after its own insert and doesn't wait on the cache write for replica rows.
- getOrCreateApp reads the race winner from the primary when its insert loses, so it no longer
  hands out a session that was never written.
- whoami runs the email check and the OIDC/taskbar/team lookups in parallel; key order unchanged.
- OIDC providers are memoized once every discovery succeeds, with one discovery fetch in flight
  per URL, and the Apple key is parsed once.
- gui-token and sync-cookie use `req.actor.user` instead of re-reading the user.
- listSessions batches app lookups. App icon probes run together, and the original-icon probe
  skips the lineage fallback.
- Default-granted permissions are answered from config before the scan cache, and exploded
  options are deduped: one less Redis read per /drivers/call.
- ACL resolves ancestors lazily (owners and full-access tokens skip them) and scans the node
  first, then the ancestors concurrently on a miss.

Filesystem
- Missing-path lineage lookups run 10 at a time: batch-upload path resolution drops from about
  1.3 s to about 40 ms per 100 new files.
- MySQL single-file inserts return the row built from the written values, falling back to the
  primary read: one less primary round trip per new file.
- Batch thumbnails are stored with one UPDATE per chunk (200 rows or 4M chars), with a per-row
  fallback.
- storage.quota.bonus listeners run concurrently next to the usage SUM, and the payload carries
  `userUuid`: quota checks and /df no longer serialize the listener reads.
- readdir cache backfill is one pipeline instead of 3xN SETEX.
- Profile reads skip the lineage fallback; the own profile is read alongside the user lookup.
- Legacy /stat runs is_empty (new `hasChildrenByParentUid`), owner and thumbnail concurrently.
- Downloads and the video proxy pipe through `pipeToResponse`: aborts release the storage stream
  at once with no extra 'close' listeners, and the video proxy no longer crashes on a mid-stream
  upstream error.
- mkdir looks up target and parent together.
- Upload completion reuses the session from the access check (status is still re-read before
  completing), and multipart signing reads it once.
- Moving a shared file to Trash skips two lookups that always returned the entry itself.

Drivers and events
- Subscription resolvers run concurrently and are folded in registration order.
- SubdomainDriver and WorkerDriver overlap independent lookups.
- The events "anyone subscribed?" check is one Redis pipeline when the region is warm.
- Handler publish overlaps the count with the size check.
- Broadcast flushes peers concurrently over a keep-alive agent, so a slow peer no longer delays
  cross-region invalidations to the others.

Sites, share and AI
- Hosted sites resolve the owner first, then the app and root together. Site config loads only
  on 404s. The hosted-app query selects only the columns it uses and moves into AppStore.
- shareFlags asks about the plan only when an item is shared by link only.
- Share listings run the count, teams and plan reads alongside their page query.
- Claude attachment cleanup no longer blocks the response.
- New util `awaitLater` for promises started early and awaited later.
2026-10-17 12:03:35 -07:00
Daniel Salazar 9ad7b99607 fix(puter-js): handle file picker cancel message (#4257) 2026-10-16 23:51:55 -07:00
Daniel Salazar 28f162ee92 fix(metering): fence settles, idempotent monthly claim, split addon consumption (PUT-2001) (#4236)
* fix(metering): pin metering writes to one replica, fence settles, idempotent monthly claim (PUT-2001)

store-kv-v1 is a last-writer-wins global table, and every region settled
its metering buffer into its local replica, so concurrent increments to the
same per-actor item from two regions could drop one side. Writes and
consistent reads of metering:* and policy:actor:* items now go to the
dynamo.home replica.

Buffered settles record each write on the claim with a fence token before
sending it, and the KV store refuses a token the item already holds, so a
re-driven claim applies its write once. Claims are serialized per counter
within a cache so one writer's tokens land in order.

The monthly charge claim is a set-if-absent of a random token, and a failed
claim write keeps its token for the retry, so a write that landed anyway
still charges exactly once.

* fix(metering): drop home-replica pinning; keep addon consumption off the purchase record (PUT-2001)

Pinning every metering write and exact read to one replica put a
cross-region round trip on credit checks near the allowance and on every
settle, and made one region's outage stall metering everywhere. The
per-item clash it guarded against is rare and undercharges at most a
flush window.

The one clash that loses money is a credit purchase and an overage settle
landing on the addons item in different regions at once. Consumption now
goes to its own item and getActorAddons sums it with what the purchase
record already holds.

* perf(metering): read addon consumption only for accounts with purchased credit (PUT-2001)

* perf(metering): settle counters through a sliding pool instead of waves (PUT-2001)

* feat(metering): consistent-read option on getActorAddons (PUT-2001)

Admin credit checks read the purchase record directly to get a consistent
read, which misses consumption now that it lives on its own item. They can
call getActorAddons with consistentRead instead. Also formats the lines this
PR added.
2026-10-16 23:49:23 -07:00
Daniel Salazar fa62b3fb76 move template menu entries into a tested helper (#4258)
Builds the "New > User templates" entries in helpers/templateMenuItem.js
and covers the label with tests, including HTML encoding of both the
extension and the name.
2026-10-16 23:45:04 -07:00
Filip Kujundžićand404oops e950a87f29 Add Gemini Omni video generation with token metering (#4235)
* Add Gemini Omni video generation with token metering

* Keep explicit Gemini video model selection within provider

* Settle reported Gemini video usage before validating output

---------

Co-authored-by: 404oops <[email protected]>
2026-10-16 23:21:48 -07:00
1cb9dd756a Add Gemini transcription with consistent usage settlement (#4234)
* feat(ai): add Gemini unary transcription

* Clarify transcription language results and omit pricing

* style(ai): keep Gemini transcription sources prettier-clean

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(ai-speech2txt): charge Gemini transcriptions that fail after Google answers

Google bills an interaction it processed whether or not it completed, so an
incomplete, failed or malformed reply was free to the caller and paid by
Puter; audio whose transcript overran the output cap could be resubmitted at
no cost. Usage is now recorded before the reply is checked, falling back to
the counted audio tokens when the reported usage can't be read.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Preserve transcript text with optional malformed annotations

* Keep speech provider spacing consistent

---------

Co-authored-by: 404oops <[email protected]>
Co-authored-by: Claude Opus 5.5 <[email protected]>
2026-10-16 23:21:47 -07:00
be16bca1d9 Keep Gemini speech instructions separate and validate speech requests (#4233)
* fix(ai): keep Gemini speech instructions separate from transcript

* Clarify Gemini speech instructions

* style(ai-tts): keep Gemini cost comment prettier-clean

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* Validate speech requests and settle reported Gemini usage

* Clarify speech settlement for unreadable responses

* Reserve the published Gemini speech token budget

* Clarify speech accounting for bounded responses

* Limit only Gemini TTS text to 3000 characters; charge overrun replies

The 3000-character text cap moves from the TTS driver into the Gemini
provider, so other providers keep their own limits.

The Gemini reply cap rises to 64 MiB, which fits the full 16,384-token
audio budget. A reply that still overruns it or times out reports no
usage, so it is charged the held budget and the error tells the caller
to split the text.

---------

Co-authored-by: 404oops <[email protected]>
Co-authored-by: Claude Opus 5.5 <[email protected]>
Co-authored-by: Daniel Salazar <[email protected]>
2026-10-16 23:21:47 -07:00
2305d50e36 Restore supported Gemini chat models and align scheduled pricing (#4232)
* feat(ai): restore available Gemini models and align pricing

* Exclude deprecated Flash Image additions and share scheduled pricing

* Bill Gemini 2.5 Pro at its current list rates

Cache reads are $0.125/M, and prompts over 200k tokens bill every rate at
the long-context tier ($2.50 in, $15 out, $0.25 cached).

---------

Co-authored-by: 404oops <[email protected]>
Co-authored-by: Daniel Salazar <[email protected]>
2026-10-16 23:21:47 -07:00
352427687e Add Gemini Argon with mode-specific limits and native usage metering (#4231)
* feat(ai): add Gemini 4 Argon with native usage metering

* test(ai): isolate Argon HTTP smoke test from optional extensions

* fix(ai): preserve Gemini native completion and refusal reasons

* Refer model specifications and pricing to the directory

* Bound unary Argon requests and reject unsupported content

* Keep Gemini tool metadata out of Claude requests

* Describe provider-specific chat timeout boundaries

* Keep Gemini extra_content off other providers; report Gemini timeouts as 504

Replayed tool calls only carry extra_content for providers that opt in
(Gemini), so OpenAI-compatible and Responses vendors don't see an unknown
field. The native Gemini client now owns its request deadline: the SDK's
own timeout aborted without a reason and read as an internal error. A
deadline timeout returns 504 upstream_timeout and leaves the route in
rotation.

---------

Co-authored-by: 404oops <[email protected]>
Co-authored-by: Daniel Salazar <[email protected]>
2026-10-16 23:21:46 -07:00
145d54cf69 Handle unavailable chat model routes and exclude batch variants (#4230)
* Handle model access denials and filter batch chat variants

* Raise a low alarm when a provider denies model access

A 403 model_not_found still returns 400 upstream_bad_request, but now
alarms at info so a lost provider entitlement gets noticed. HttpError
gains an alarmSeverity option the terminal alarm gate honors.

---------

Co-authored-by: 404oops <[email protected]>
Co-authored-by: Daniel Salazar <[email protected]>
2026-10-16 23:21:46 -07:00
Reynaldi ChernandoandClaude Opus 5.5 471b900462 remove private access routing from docs (#4252)
Co-authored-by: Claude Opus 5.5 <[email protected]>
2026-10-16 20:14:21 +07:00
Daniel Salazar 2b254b8892 fix: misc bugs (#4251) 2026-10-15 23:49:17 -07:00
Neal Shah 2a15afc570 Resolve PUT-2059 (#4250) 2026-10-16 01:45:56 -04:00
dependabot[bot] f9860b036b chore(deps-dev): bump prettier from 3.9.9 to 3.9.10 (#4240)
Bumps [prettier](https://github.com/prettier/prettier) from 3.9.9 to 3.9.10.
- [Release notes](https://github.com/prettier/prettier/releases)
- [Changelog](https://github.com/prettier/prettier/blob/main/CHANGELOG.md)
- [Commits](https://github.com/prettier/prettier/compare/3.9.9...3.9.10)

---
updated-dependencies:
- dependency-name: prettier
  dependency-version: 3.9.10
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-15 14:46:31 -07:00
dependabot[bot] 76e1f79b5e chore(deps): bump smtp-server from 3.19.17 to 3.19.21 (#4242)
Bumps [smtp-server](https://github.com/nodemailer/smtp-server) from 3.19.17 to 3.19.21.
- [Release notes](https://github.com/nodemailer/smtp-server/releases)
- [Changelog](https://github.com/nodemailer/smtp-server/blob/master/CHANGELOG.md)
- [Commits](https://github.com/nodemailer/smtp-server/compare/v3.19.17...v3.19.21)

---
updated-dependencies:
- dependency-name: smtp-server
  dependency-version: 3.19.21
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-15 14:43:54 -07:00
dependabot[bot] 943a3427c6 chore(deps-dev): bump @types/busboy from 1.5.4 to 1.5.5 (#4241)
Bumps [@types/busboy](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/busboy) from 1.5.4 to 1.5.5.
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/busboy)

---
updated-dependencies:
- dependency-name: "@types/busboy"
  dependency-version: 1.5.5
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-15 14:38:42 -07:00
dependabot[bot] 38469a3615 chore(deps): bump libphonenumber-js from 1.13.14 to 1.13.15 (#4239)
Bumps [libphonenumber-js](https://gitlab.com/catamphetamine/libphonenumber-js) from 1.13.14 to 1.13.15.
- [Changelog](https://gitlab.com/catamphetamine/libphonenumber-js/blob/master/CHANGELOG.md)
- [Commits](https://gitlab.com/catamphetamine/libphonenumber-js/compare/v1.13.14...v1.13.15)

---
updated-dependencies:
- dependency-name: libphonenumber-js
  dependency-version: 1.13.15
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-15 14:35:15 -07:00
dependabot[bot] a4d4a5595e chore(deps-dev): bump vite from 8.1.5 to 8.1.6 (#4238)
Bumps [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite) from 8.1.5 to 8.1.6.
- [Release notes](https://github.com/vitejs/vite/releases)
- [Changelog](https://github.com/vitejs/vite/blob/v8.1.6/packages/vite/CHANGELOG.md)
- [Commits](https://github.com/vitejs/vite/commits/v8.1.6/packages/vite)

---
updated-dependencies:
- dependency-name: vite
  dependency-version: 8.1.6
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-15 14:35:07 -07:00
dependabot[bot] de69269b10 chore(deps): bump uuid from 14.0.2 to 14.0.3 (#4237);'''"????????????????????
Bumps [uuid](https://github.com/uuidjs/uuid) from 14.0.2 to 14.0.3.
- [Release notes](https://github.com/uuidjs/uuid/releases)
- [Changelog](https://github.com/uuidjs/uuid/blob/main/CHANGELOG.md)
- [Commits](https://github.com/uuidjs/uuid/compare/v14.0.2...v14.0.3)

---
updated-dependencies:
- dependency-name: uuid
  dependency-version: 14.0.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-15 13:25:49 -07:00
Neal Shah 8faff74793 PUT 2057 and PUT 2068 (#4245) 2026-10-15 02:02:00 -04:00
Neal Shah e5ca303c50 PUT-1903 (#4226) 2026-10-14 20:29:55 -04:00
Juan Fernando Castro c6c0dd5e29 fix: bound how many accounts a team owner can provision (#4223)
* feat: bound provisioning by what an owner may mint, not only what is live

Seat and team caps express how much may exist at one time. This adds the
other half of the rule: how much an owner may create over a window, and the
checks a provisioned account answers to.

- Provisioning puts each seat through `SignupService.runSignupGate`, the same
  extension-owned gate a signup answers, and refuses what it blocks. It fails
  closed, since a check that could not run is not a check that passed.
- A per-owner budget over a rolling window, sized to every seat the owner may
  hold live across every team they may own, times the churn allowed on top.
  It is spent on creation and is not returned.

Keyed on the owner's uuid so a recycled id cannot inherit someone else's
spend, and charged after validation so a taken username costs nothing.

* refactor: tighten the provisioning gate and budget after review

- The gate runs before the seat-cap lock. A listener has no deadline and the
  lease is ten seconds with nothing to renew it, so the lock must not be held
  across one.
- `failOpen: false` rethrows rather than returning false, so a backend outage
  is answered 503 instead of escaping the handler.
- An address goes through `validateEmail` first, as it does on every other
  path that creates an account.
- The gate always runs; a request only enriches what it is told, so one being
  absent is no reason to skip it.
- The gate is told it is looking at a seat, whose address is optional.
- A refusal carries the gate's own code and request code, as a blocked signup
  does, so support can correlate it.
- Restored `provisionAccount`'s docblock, which an insert had displaced.

Tests: the spy restores in `finally`, and the cap overrides are taken after
the fixtures, so one failure cannot cascade through the file.

* fix: show a provisioning refusal as a rule, not a failure

The console special-cases the seat cap and sends everything else to
"Something went wrong", which reads as a fault rather than an answer. The
provisioning budget refuses the same way the cap does, so it is presented
the same way.
2026-10-14 11:54:35 -04:00
Juan Fernando Castro e32bfa839d fix: count one app open per account per app per hour (#4152)
* fix: count one app open per account per app per day

`app_opens` is read two ways at once: `open_count` is a row count and
`user_count` is distinct accounts. A repeated open moves the first
without the first moving the second, and the incentive report prices
opens.

An open is now marked as it is recorded and skipped for the rest of the
day, before both the row and the event ClickHouse ingests, so the two
stores agree. The marker is set only when absent, so the first open of
the day wins a race; an unreachable marker counts the open, since
counting twice beats not counting.

* fix: count one app open per account per app per day, at the count

Dropped the row-suppression approach. `app_opens` is not only a counter:
`getRecentAppOpens` orders the desktop's recent list by `MAX(_id)`, and
`app.opened` is the socket event other tabs reorder on. Suppressing the
repeat row froze both at the first open of the day.

Every open is recorded as before. `open_count` counts each account once
per day instead, which is the figure `user_count` is read against.

`countDistinctUserDays` folds the pair into one number because
`COUNT(DISTINCT a, b)` is MySQL-only, and `dayBucket` covers MySQL's
decimal division. ClickHouse has uniqExact over the tuple.

Note the figure's meaning changes at deploy: a period spanning it mixes
rows with account-days, so a comparison across that point is not one.

* fix: count one app open per account per app per hour

Review feedback: a day is too long a window for the repeat to collapse
into. The period is now an argument rather than a baked-in 86400, held in
one constant on AppStore beside the other store policy numbers, so the SQL
and ClickHouse sites cannot drift and a future change is one line.

The fold `COUNT(DISTINCT user * N + bucket)` is only injective while the
bucket index stays under N. Hourly buckets are 24x the day index and pass
the old 1000000 in 2084, so N moves to 10000000.

Buckets are clock hours, not a rolling window, so two opens a minute apart
either side of the hour still count twice — tested, and the same property
the day version had at UTC midnight.

The detailed-stats fixture gave its repeat open a 60s offset, which under
hourly buckets would have counted twice whenever the suite ran in the first
minute of an hour. It now shares a timestamp with the open it repeats.
2026-10-14 11:54:17 -04:00
Daniel Salazar ff815f730d fix: harden token delivery, session revokes, app identity and fs scoping (#4229)
- relay sign-in popups take the opener origin from the referrer only
- session revokes read the primary, batch per user, and tombstone caches in every region
- wisp relay tokens bound to the minting session; verify rechecks the account
- hosted origins keep app identity with the site owner; deleted app uids recorded in deleted_apps
- subdomain cache tombstoned on delete, update, reclaim and app/account cascades
- email lookups require an exact address match after the SQL lookup
- workers deploy only from files the deployer owns
- share_app_data:false enforced at ACL time, plus copy/move/rename/share/recursive listing
- kv remove returns no value without read; kv:expire op grant usable
- fs errors name the entry instead of the owner's path
- shared-by-me drops rows a delegate no longer has authority over
- private apps can't move index_url onto another user's origin
- GUI: default apps saved and launched by uid; save picker accepts only copy/move; permission dialog waits for focus; Publish as Worker hidden on others' files
2026-10-14 06:56:44 -07:00
Juan Fernando Castro 84aaf005ec fix: give every route behind the user-protected gate a password budget (#4150)
* fix: give every route behind the user-protected gate a password budget

Five of the six routes behind the gate declare a per-user hourly bucket
for the password check. The sixth, delete-own-user, declared none.

The bucket moves into the gate itself, so it covers every route behind
it and a new one cannot be added without one.

It is charged by the outcome rather than by the request: a wrong answer
spends one and a right answer spends nothing, so a caller who knows the
password never meets the limit.

* fix: the review's findings on the password budget

Reading the budget and spending it on the outcome bounded rounds rather
than attempts: everything already in flight read the same budget and was
answered before any of it was spent. The charge lands before the
comparison now, so a burst is bounded by the same number as a sequence.

The key gained a route component. One bucket across all six meant ten
mistyped passwords on one route closed the other five for the hour,
including the one an account owing its team's 2FA has to reach.
2026-10-14 09:52:23 -04:00
Daniel Salazar f6a1e54642 fix(fs): keep S3 read stream errors from crashing the process (#4227)
getObjectStream now attaches an 'error' listener to every body it returns.
A body that fails before its consumer attaches (socket timeout, reset, or a
checksum mismatch on a truncated read) used to emit 'error' with no listener
and exit the process, failing every in-flight request on the task. It now
logs a warning; consumers that attach later still receive the error.
2026-10-13 17:42:28 -07:00
Filip Kujundžić d211a25c6d Merge pull request #4218 from HeyPuter/codex/put-1851-deepseek-peak
Protect direct DeepSeek peak-only pricing
2026-10-14 00:54:11 +02:00
404oops 2e1bf86d2b test(ai): protect direct DeepSeek peak-only pricing 2026-10-13 20:44:00 +02:00
404oops 8c31670453 Align ZAI modality metadata with source conventions 2026-10-13 20:43:58 +02:00
404oops 16d0ce8b3e Advertise supported image input for ZAI Flash models 2026-10-13 20:43:58 +02:00
404oops 8c64f54a87 Remove ignored-option paragraph from speech reference 2026-10-13 20:43:54 +02:00
404oops 7abd7b4417 Remove unsupported speaker reference option 2026-10-13 20:43:54 +02:00
404oops 29e0e1e2aa Clarify speech catalog units and cover ignored transcription options 2026-10-13 20:43:54 +02:00
404oops 4857898af4 Assert finite ElevenLabs catalog prices 2026-10-13 20:43:54 +02:00
404oops 526a250c95 Correct speech catalog pricing and legacy option declarations 2026-10-13 20:43:54 +02:00
404oops 8c18c45e63 feat(ai): refresh verified provider pricing catalogs 2026-10-13 20:43:43 +02:00
404oops 124f95a5f9 Label provider catalog fixtures as test data 2026-10-13 20:43:43 +02:00
Neal ShahandClaude Opus 5.5 4401b4c4dc Fix worker test (#4156)
* PUT-2012 fix broadcast service drops

* Revert pre-commit lint churn on files merged from main

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* fix test bug

* Remove other test

---------

Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
2026-10-13 14:10:10 -04:00
Neal Shah 62270ff9aa Delete worker sessions when workers are deleted, don't treat sessions from deleted workers as still valid. (#4163) 2026-10-13 12:43:45 -04:00
404oops ddafc2d04c fix(ai-chat): normalize remote catalogs and conditional pricing 2026-10-13 18:25:36 +02:00
Daniel Salazar b52fc01712 perf: cut godmode app launch latency (cached token renewal, launch reuse) (#4221)
* perf(auth): renew godmode app tokens from cache instead of the primary

Every godmode launch read its token row from the primary and rewrote its
expiry, a few cross-region round trips per launch far from the primary.

- A cache pointer names the row per (user, web session, app). The row is
  read through the uuid cache token auth uses and re-checked (owner,
  session, app, revoked, expired, max age); anything else falls back to the
  primary read. Revoking the row drops the pointer.
- The expiry write is skipped when it would move by under an hour. A
  renewal that hits a revoked or lapsed row mints a new one instead.
- The response carries the row's actual expires_at.

* perf(gui): reuse a just-minted godmode token for repeat launches

A launch of a godmode app reuses the token this page minted for it within
the last ten minutes under the same desktop session, instead of asking
again. The cache is dropped when an app reports its token dead, after a
session revoke from the session manager or an uninstall, and across tabs
via a BroadcastChannel that carries token row ids, never tokens. A mint in
flight during a drop is not remembered.
2026-10-13 08:53:55 -07:00
Himanchal Kaushale f37cb22d8a fix(gui): resize app content above mobile keyboard (#4165)
Add interactive-widget=resizes-content to the mobile viewport meta tag
so the layout viewport shrinks when the soft keyboard opens. The app
window's height chain resolves against the layout viewport, so the
iframe folds up above the keyboard instead of clipping behind it.
2026-10-13 08:53:44 -07:00
Reynaldi Chernando 948772165d Various docs fixes and AGENTS.md update (#4211) 2026-10-13 17:29:46 +07:00
Neal Shah b8bb6292d6 Fix webdav locks issue and tighten GUI ipc rules for setWindow* (#4207) 2026-10-12 17:39:09 -04:00
Filip Kujundžić 542ca9ee1c Merge pull request #4202 from HeyPuter/filipkujundzic/speech-contract-fixes
fix(ai-speech): honor ElevenLabs output formats, voice IDs and URL audio
2026-10-12 16:21:33 +02:00
404oopsandClaude Opus 5.5 8e92bf6803 fix(ai-speech): honor ElevenLabs output formats, voice IDs and URL audio
- TTS sends output_format as the query parameter ElevenLabs reads; the body
  field was ignored, so every request came back as MP3. Unknown formats are
  rejected before credits are reserved, and the OpenAI-style names mp3, opus,
  pcm and wav map to native formats.
- Voice IDs go straight to synthesis. The library listing omits voices that
  still synthesize, the default 21m00Tcm4TlvDq8ikWAM among them, and runs to
  9 MB per call. Names resolve through /v2/voices search, and premade voices
  ("Bella - Professional, Bright, Warm") also match their short name.
- speech2speech accepts HTTPS audio URLs through secureFetch, as documented.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-12 12:40:43 +02:00
404oopsandClaude Opus 5.5 b61448c8b7 feat(ai-image): add Gemini Nano Banana 2.1
Catalog gemini-nano-banana-2.1 (aliases google/gemini-nano-banana-2.1 and
nano-banana-2.1) at Google's standard rates: $1.50/1M input, $7.50/1M text
and thinking output, $30/1M image output, with 1K/2K/4K tiers at
1120/1680/3780 image tokens. 3.1 Flash Image stays the provider default.

Pre-flight credit checks assumed 50 text/thinking output tokens per image
for every Gemini model; live samples run 270-1122 depending on the model, so
each model now carries its own estimate.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-12 12:39:49 +02:00
404oops f9cde23937 Merge branch 'filipkujundzic/put-1954-elevenlabs-tts-passes-voice-names-through-as-voice-ids-so' into filipkujundzic/put-1907-ai-streaming-buffering-and-error-shape-cleanup
# Conflicts:
#	src/backend/drivers/ai-chat/utils/inlineImages.test.ts
#	src/backend/drivers/ai-chat/utils/inlineImages.ts
#	src/backend/drivers/util/fileInput.ts
2026-10-12 12:39:35 +02:00
404oops 94e84a2819 Merge branch 'main' into filipkujundzic/put-1954-elevenlabs-tts-passes-voice-names-through-as-voice-ids-so 2026-10-12 12:36:26 +02:00
404oopsandClaude Opus 5.5 f0b55e7605 docs(ai): trim listModels return value and AI overview per review
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-12 12:33:05 +02:00
Daniel Salazar d8f9aff3dd fix: follow-ups to #4161–#4203 (rate limits, broadcast, AI routing and billing, fs subtree walks) (#4205)
* fix(rate-limit): count the newest KV rate rows first

The KV backend's rate check, peek and concurrent acquire list the key's
rows with a limit + 1 cap that the store applies before dropping expired
rows, so unswept rows from past windows (oldest keys) could fill the fetch
and the live count read as zero. They now list newest first.

* fix(broadcast): split webhook flushes under a byte budget, per-peer backlogs

Flushes go out in bodies of at most `broadcast.outboundMaxBodyBytes` (default
4 MiB) instead of one POST, and each peer keeps its own backlog of what it
missed, capped by count and four bodies' worth of bytes, oldest dropped first,
so one failing peer can no longer grow every peer's body past their 50 MB limit.

* fix(ai-chat): classify dropped upstream connections as provider failures

A connection reset or refusal (SDK APIConnectionError, Mistral's
ConnectionError, an undici reset on the cause) carries no status, so a chain
of them fell through to 500 internal_error. Attempts now record
connectionFailed; a chain of only resets and 5xx answers 400
upstream_provider_unavailable, and a reset counts as an upstream signal in a
mixed chain.

* fix(ai-chat): keep SDK retries on Anthropic count_tokens

The Claude client is built with SDK retries off so the driver can fall back,
but count_tokens has no fallback route. It now passes sdkRetryOptions, like
the OpenAI moderation call, so a transient 429 or reset is retried.

* fix(ai-chat): mark a route unhealthy for 60 s after a transient failure

With SDK retries off, one 5xx, 429, timeout or dropped connection kept a
route marked for 10 minutes, sending that model's traffic to resellers. Those
failures now mark the route for 60 s; bad credentials, an exhausted account
and other route-level failures keep 10 minutes.

* fix(metering): don't memoise spent addons inside the force-exact window

The spent-addons memo could cache a pre-grant record read from a lagging
replica right after a credits-changed announcement, so overage skipped the
new credit and requireCredits kept answering 402 for up to 15 s. A spent
record is now only cached once the actor's force-exact window has ended.

* fix(share): send the digest entries a flush claimed when another take fails

Digest entries are taken concurrently, and a single rejected take threw
away every entry the other takes had already removed from KV. The flush
now logs the failed takes and sends what it claimed; an entry whose take
failed stays queued for a later flush.

* fix(fs): announce an overwriting move's replaced entry only when it took its name

/move and v2 /fs/move looked up the entry at the destination name before
moving and always announced its removal. A move into someone else's Trash
renames instead of replacing, so the owner's desktop hid a live entry and
legacy /move reported it as `overwritten`. Both routes now pass the target
on only when the moved entry kept the name it was looked up by.

* fix(ai): name the vendor in upstream error fields from media providers

upstreamFetch now takes the provider id plus a display label, and
UpstreamResponseError carries the id. translateProviderError copies a
string `provider` off the error into `fields`, and the BytePlus video and
Cloudflare image errors set one, so upstream_auth_failed and
upstream_credits_exhausted alarms say which vendor failed again.

* fix(teams): delete a provisioned seat whose home folders could not be created

provisionUser logs a failed default-folder creation and still returns the
account, so provisionAccount went on to add the membership, issue a password,
mail the seat and emit team.account.created for an account with no home. It
now cascade-deletes the new user and returns 500 internal_error before any of
that.

* fix(ai-chat): load model catalogs without holding up server start

onServerStart awaited every gateway catalog (up to its 10s fetch timeout)
while HTTP was already listening, so drivers started after it had no
providers yet and chat requests found an empty model map. The first build
now runs in the background; complete(), models() and token counting wait
on it instead of failing.

* fix(ai-chat): serve direct models while gateway catalogs load at boot

The first model-map build now publishes a fresh map as each provider's
catalog lands, so static catalogs are routable at once. complete() and
token counting wait for the full first build only when the model, or the
provider asked for, isn't in the map yet. Refreshes still swap once.

* fix(ai-chat): split gateway-billed cost across input and output

A completion billed at the cost OpenRouter, Infron or Neuralwatt reports
was quoted to `ai.prompt.cost-calculated` as all input and no output. The
ledger still records one `billedUsage` line; the reported per-key costs now
spread that bill over the usage keys at the catalog's relative rates, in
whole µ¢ that sum to it. Neuralwatt energy units take no share.

* fix(ai-image): hold Replicate credits for the whole prediction

The output and input-surcharge holds took the default 10-minute TTL, the
same as the prediction window alone, so a slow prediction could outlive
them once creation and input measurement were added. They now last
15 minutes, like the video job holds.

* feat(ai-chat): report what a completion was charged as usage.cost

Chat usage (the non-stream result and the stream's final usage line) now
carries `cost`: the µ¢ the ledger records for the request after the cost
factor, from the request's recorded charges and the gate's factor lookup.
`usd_cents` is unchanged. A hooked recording now copies the usage object
first, so keys the driver adds later never land in the ledger.

* fix(ai): charge streamed speech once its first byte arrives

Per-character TTS and the voice changer recorded their charge when the
upstream's response headers arrived, so a body that failed before any audio
was billed in full. A shared relay now charges on the first byte, or when
the caller hangs up, and charges nothing for an upstream that fails or ends
before sending audio.

* fix(ai): release credit holds only after the request's usage lands

A hold was released as soon as its operation returned, while the usage it
stood for was still being written (behind the cost-factor lookup), so a
concurrent request briefly saw neither. AI metering now tracks the
request's usage writes and its holds (chat included) release once those
settle, without delaying the response; a streamed charge chains its own.
A failed write still releases the hold, and the hold's TTL backstops it.

* test(ai-chat): wait for a stream's credit hold to come back after its usage lands

* perf(fs): page subtree delete and copy by parent link, not by depth over the path range

Each page of a recursive delete or copy sorted the whole remaining subtree by a
computed depth, so a walk cost grew with the square of its size. It now pages
the children of up to 100 sibling directories at a time by parent_uid, depth
first, keyset on (parent_uid, id), so a page reads only its own rows; rows with
no stored path are reached too. SQLite gets the parent_uid index the others have.

* fix(fs): find children by parent_uid in the empty check and supplementary-name lookup

Rows the old backend wrote carry parent_uid and no parent_id, so a non-recursive
delete of a directory holding only such rows passed the empty check and orphaned
them, and on MySQL the supplementary-character child lookup missed them, leaving
their subtrees out of listings, counts and renames. Both now key on parent_uid;
the empty check still probes parent_id too, which the delete cascade follows.

* docs(fs): drop the single-POST note from the entry broadcast cap
2026-10-12 01:10:54 -07:00
Daniel Salazar b88aa55b24 fix: drop moved descendants from the entry cache; read cursor=null as the first page (#4204)
Renaming or moving a directory rewrote its descendants' paths but left their
cached copies (by id, uuid and old path) live for up to 60 s. A lookup of an
old path kept resolving, and a write beneath one landed at a path its parent
no longer had. updatePathPrefixForUser and renameUserHome now drop those keys,
here and, within the broadcast budget, in peer regions.

puter.js sent `cursor=null` for `listShared`/`listSharedByMe({ cursor: null })`,
which decoded to null and failed every keyset listing with a 500. decodeCursor
reads it as no cursor, and the SDK no longer sends it.
2026-10-12 00:19:11 -07:00
Daniel Salazar ff13edc19a Merge branch 'main' into filipkujundzic/put-1954-elevenlabs-tts-passes-voice-names-through-as-voice-ids-so 2026-10-12 02:21:00 -04:00
Daniel Salazar f07a786b41 fix: reserve storage before creating directories on signed write starts (#4161)
startBatchWrite and startWrite created directory items and missing
parents before (or alongside) the storage reservation, so an over-quota
start paid for every directory write, left the directories behind, and
waited on them before returning 413. File targets are now resolved and
storage reserved first; directories are created only after, and the
reservation is released if that fails.
2026-10-11 22:14:57 -07:00
Daniel Salazar 21131ad3d6 fix(kv): spread upload sessions across system KV partitions (#4162)
Upload sessions shared the system namespace's single partition key. They
now go to one of 16 system partitions hashed from the session id, via a
new system-actor-only `systemPartition` KV option; partitions are never
cached and emit no mutation events. Reads fall back to the unpartitioned
key for sessions written before this (max session lifetime is one hour).
2026-10-11 22:14:11 -07:00
Daniel Salazar 38c986927c fix(fs): page subtree walks over supplementary-character names on MySQL (#4203)
listDescendantsInOrder, which drives paged recursive delete and copy, still
used a bare `path LIKE` range, so on MySQL a child whose name starts with a
supplementary character (and its subtree) was skipped: delete left it to the
parent_id cascade and copy dropped it. It now ORs the subtree patterns like the
other subtree queries, paging each pattern group with the same seek and
merging the pages in order.
2026-10-11 22:11:39 -07:00
Daniel Salazar 590c87c00a refactor(stores): one row cache and keyset page helper, batched cache round trips, dead store methods removed (#4191)
* refactor(stores): one read-through RowCache for UserStore and AppStore

UserStore and AppStore each carried the same multi-key row cache: key
builder, cache read, tombstone-guarded backfill, broadcast refresh,
invalidate and tombstone-on-delete. They now share RowCache on PuterStore,
which also single-flights concurrent misses for one key and batches
getByIds/getByUids: one pipelined tombstone check for the loaded rows and
one pipelined backfill instead of per-row awaits. Cache keys and TTLs are
unchanged.

* perf(apps): invalidate cached app lists by generation, not by tracker scan

Every app write SMEMBERS'd a global set of every cached list key, parsed
each key back into its filters and deleted the matches. List pages are now
keyed by their owner's generation token (or a shared one for unscoped
lists), and a write replaces the tokens it affects: O(1) per write, and a
page loaded before a write can no longer be cached after it.

* refactor(stores): one keyset page helper for store listings

Thirteen store listings each cut a LIMIT n+1 read to a page and sealed the
last row's position, and each opened an id cursor its own way. They now use
keysetPage() and openIdCursor() from util/pagination. Cursor payloads and
sealing are unchanged; a cursor without a usable id still fails with 400 in
ShareStore and still restarts the listing elsewhere (strict option).

* refactor(stores): delete store methods nothing calls

AppStore.existsByIndexUrl, OIDCStore.unlinkByUserId,
TeamStore.listByOwner/listActiveSeatUuids, SubdomainStore.getByDomain/
listByDomain, FSEntryStore.ensureDirectoriesForUser,
ShareStore.listByIssuer/listByRecipientEmail/deleteByRecipientEmail/
getDailyShareCount/listByHolderAndFsentries and
PermissionStore.invalidateScanCache/deleteUserUserPermsByPermissionPrefix
had no production callers. ShareStore.listByFsentries was a copy of
listReaching; its callers use the latter.

SubdomainStore SMEMBERS'd a per-user prefix-list tracker on every write and
broadcast its deletion, but nothing ever wrote that set or cached a prefix
listing. The reads are gone.

* refactor(stores): shared escapeLike and bumpGeneration helpers

escapeLike replaces the LIKE escaping copied into PermissionStore (twice),
ShareStore and FSEntryStore, and now also guards SubdomainStore's prefix
filters, which passed caller prefixes into LIKE unescaped.

bumpGeneration sends INCR and EXPIRE in one pipeline instead of two awaited
commands in PermissionStore, EventSubscriptionStore and PresenceStore.

* refactor(stores): read affected rows off WriteResult

Every database client returns affectedRows and anyRowsAffected, so the
`affectedRows ?? changes ?? 0` fallbacks in ShareStore, NotificationStore,
UserStore and UserBlockStore read those directly.

* perf(stores): batch flat-permission KV deletes and durable cache drops

PermissionStore deleted each flat KV entry with its own kv.del; one
kv.batchDel now covers the set. DurableSubscriptionStore's suspend and reap
dropped each row from the cache separately; EventSubscriptionStore gains
dropDurables, which drops them per owner in one pass, and dropDurable calls
it. suspend keeps one conditional UPDATE per row, which is how concurrent
settles learn which rows each of them suspended.

* refactor(fs): fold FSEntryStore's duplicate getters, sort maps and cache keys

- getEntryByUuid and getEntryById share one by-column getter.
- listChildren and listChildrenPage read one sort table; the offset listing
  now coalesces NULL modified/size like the cursor listing, so both order
  NULLs the same way on every engine.
- Entry cache keys come from one builder instead of hand-built strings.
- getEntriesByIds, getEntriesByPaths and the per-user path reader fetch
  their cache hits in one pipelined round trip instead of a GET per key,
  through readCacheValues, which RowCache.readMany now uses too.

* fix(apps): cache the row a lost createFromOrigin race reads from the primary

Keeps AppStore's private #writeCache as a RowCache write, so code that
still calls it (#4166's read-back) loads in either merge order.
2026-10-11 22:07:53 -07:00
Daniel Salazar ed22e2b285 docs: tighten contributor and architecture guides (#4194) 2026-10-11 22:07:02 -07:00
Daniel Salazar 9030b45459 refactor(ai-chat): one OpenAI-compatible provider, one pricing path, cached catalogs (#4192)
* perf(ai-chat): cache gateway catalogs with one remote-catalog helper

cachedRemoteCatalog serves a fetched catalog for 15 minutes, shares one
in-flight fetch between callers, abandons a fetch after 10 s, remembers a
failure for 30 s and keeps the last good value through a failed refresh.

Together, OpenRouter, Infron, Neuralwatt (models and quota) and Ollama use it
in place of their kv-singleton caches. The cached value is the converted
catalog, so a completion no longer re-coerces the whole gateway catalog
(OpenRouter did it twice per request). Ollama now refreshes instead of caching
its first non-empty list forever, and Together's catalog failure no longer
rejects models().

* refactor(ai-chat): build every chat SDK client with retries off

sdkClientOptions() gives each OpenAI, Groq, Together and Anthropic client
maxRetries: 0 and an explicit timeout. The SDK default of two retries, each up
to the full timeout, stacked under the driver's three-route fallback chain; a
failed attempt now goes straight back to the driver, which marks the route
and moves to the next provider for the model.

Timeouts keep what each provider already had: ten minutes, Groq's SDK default
of one minute, and Claude's 10 m 0.6 s. Non-streaming Mistral calls, which had
no timeout once a cancellation signal was passed, are bounded at ten minutes.
The Azure providers are left for after the open Azure fixes land.

* refactor(ai-chat): meter every OpenAI-family completion through one pricing path

meterChatUsage records a completion's usage at buildCostsOverride's rates, or
as one billedUsage line when the gateway reports what it billed, and returns
the per-key costs it recorded. Providers hand those to the driver through the
usageCosts channel, now carried by the Chat Completions handlers too, so the
usd_cents and cost-calculated figures equal the ledger. The driver's own
#computeCost is replaced by pricing.ts usageCost, which shares the rate
resolution with buildCostsOverride.

Billing changes, all toward what the upstream charges:
- DeepSeek, Moonshot, Alibaba, Groq, MiniMax, Z.AI, Hoonify, BytePlus and the
  Neuralwatt catalog fallback took cached reads out of prompt_tokens nowhere,
  so a cached token was billed at input + cache rate. splitCachedPrompt bills
  it once, at the cache rate; usage.prompt_tokens on those routes now excludes
  cached reads, as it already did for OpenAI, Gemini and Meta.
- Catalog entries that listed cached_tokens: 0 (38 Alibaba, 5 Groq) relied on
  that double count to bill cached reads at the input rate. The zero is
  dropped, so they keep exactly that price through the input-rate fallback.
  Neuralwatt models with no published cache price likewise.
- An unpriced key falls back to the input/output rate instead of NaN (0), e.g.
  OpenRouter cached reads on its no-cost fallback path.
- A gateway-reported cost of 0 records 0 µ¢ rather than 1 µ¢.

Claude, Azure and xAI keep their metering; xAI's cached-token handling waits
on the open Grok decision.

* refactor(ai-chat): one OpenAICompatProvider for the OpenAI-compatible vendors

DeepSeek, Moonshot, Alibaba, Groq, MiniMax, Z.AI, Hoonify, BytePlus, Together,
Gemini, Meta, Ollama and the complete() paths of OpenRouter, Infron and
Neuralwatt were copies of one Chat Completions call. Each is now an
OpenAICompatOptions entry (client, catalog, metering prefix, wire-id prefix,
max-token param and default, passthrough fields, openAICompatParams scope,
message cleanup, image inlining, stream usage, context-length retry) plus the
hooks its API needs: prepareMessages (DeepSeek, Groq, Gemini), vendorParams
(MiniMax, Hoonify, Z.AI, BytePlus, Meta, OpenRouter, Infron, Neuralwatt),
meteredUsage (Gemini and the gateways) and reportedUsage (Neuralwatt). xAI is
left until the open xAI fix lands.

Deliberate changes:
- Tools go through make_openai_tools for every vendor (six did, ten sent them
  raw): Anthropic-only tool extras and server tools no longer reach these
  upstreams, and an empty tool list is omitted.
- Model lookup is one indexed, case-insensitive pass per catalog for every
  vendor (Together and Neuralwatt already ignored case).
- Per-provider console logging of upstream errors is gone; the driver logs
  every failed attempt.
- Every vendor honours a configured apiBaseUrl.

tool_choice forwarding stays as each vendor had it (MiniMax, Hoonify, Z.AI
auto-only, BytePlus, Meta auto-only, OpenRouter); widening it needs each
vendor's support confirmed.

* perf(ai-chat): resolve the model once in the driver and keep its map fresh

The driver hands each provider its own catalog entry for the route it chose,
so providers stop re-resolving the id per request; the OpenAI-compatible
providers only look an id up (indexed, case-insensitively) for a direct
caller, and an unknown id is a 400 instead of silently becoming the default
model. Gateways with mixed-case ids (Infron, Neuralwatt) used to miss the
driver's lowercased id and fall through to that default.

The model map is now built from every provider's catalog in parallel, with a
failing catalog left out rather than aborting the build, swapped in whole,
awaited in onServerStart, and rebuilt every 15 minutes so gateway catalog
changes reach routing without a restart.

* refactor(ai-chat): drop provider surface and OpenAIUtil code nothing uses

- list() leaves IChatProvider and every provider that isn't Azure or xAI
  (both wait on the open fix for those files); only tests called it.
- OpenAI Chat, OpenAI Responses and Meta no longer inline puter_path parts:
  the driver does it before every attempt, so theirs found nothing to do. The
  stores and FS service they took for it go too.
- providers/ChatProvider.ts had no subclasses.
- OpenAIUtil: one hoistToolUseBlocks for the Chat and Responses message
  passes (Responses keeps canonical_id), and the dead create_usage_calculator,
  unused Responses deviations and constant mode/toolblock are gone.

checkModeration stays on IChatProvider: the open Azure moderation fix calls it
through that interface.

* refactor(ai-chat): register keyed providers from one table

Sixteen near-identical blocks in #registerProviders, with three constructor
argument orders, become a { name, configKey, Class } table read by one loop;
every entry takes (config, metering) and gets apiBaseUrl when configured.
Claude, Azure, OpenAI (sibling wiring), Ollama (enabled unless turned off) and
the fake provider stay explicit. Gemini's constructor takes config first like
the rest.

* fix(ai-chat): send Ollama the model name it listed

Ollama catalog ids are ollama:ollama/<name>, built from /api/tags. Only the
ollama: part was stripped, so the local server was asked for ollama/<name>, a
model it doesn't have. Strip the whole ollama:ollama/ namespace.

* fix(ai-chat): keep SDK retries on the OpenAI moderation call

The completion clients run with SDK retries off so the driver can fall back.
A moderation check has no fallback, so it passes the SDK's default two
retries per request.
2026-10-11 22:06:18 -07:00
Daniel Salazar f3134b2530 refactor(ai-media): shared upstream fetch, provider registry and bounded inputs for AI media drivers (#4188)
* refactor(ai): route TTS, STT and voice-changer REST calls through upstreamFetch

upstreamFetch bounds a provider call with AbortSignal.timeout and throws an
UpstreamResponseError carrying the upstream status and code, which the
/drivers/call boundary maps like any SDK error. Replaces five hand-written
status translators (ElevenLabs TTS, xAI TTS, Speechify, xAI STT, voice
changer); caller-facing messages are now sanitized by the shared mapper.

* perf(ai): bound input image fetches and share one data-URI parser

fetchImageBytes now streams under a byte cap (30 MB by default) with a
30-second timeout, via a readBoundedBody helper shared with loadFileInput's web
branch and Replicate's input measuring. assertInputImagesShape caps
input_images at 16. Chat image inlining uses fetchImageBytes instead of
buffering with no timeout. fileInput, imageInput and mediaParts parse data URIs
through one parseDataUri.

* refactor(ai-tts): one per-character metering path for TTS providers

meterPerCharacter holds text length x price, runs the synthesis and meters the
same amount once it succeeds; characterCostReport builds the per-character
cost lines. Both sit on a new withAiCreditHold (reserve, run, release) that
Gemini TTS uses directly. SAMPLE_AUDIO_URL is defined once. The TTSProvider
base class is gone: its methods were stubs every provider overrode, and only
Polly read its untyped providerConfig.

* refactor(ai): one ProviderRegistry and key reader for the AI media drivers

ProviderRegistry holds a driver's provider catalog (ids, default, aliases) and
its configured instances: alias resolution, the not-found 400, default
fallback and parallel 'all' listing. TTS, speech-to-text, OCR and the voice
changer route through it. readProviderKey reads apiKey, secret_key, api_key and
key in that order for every AI media driver, so providers.xai.secret_key now
enables xAI TTS as it already did xAI image and speech-to-text. ElevenLabs
caches its voice list for 10 minutes, as Polly does.

* perf(ai): stream TTS and voice-changer audio instead of buffering it

OpenAI, xAI and ElevenLabs TTS and the voice changer hand the upstream body
straight to the caller through upstreamBodyStream; their costs are known
before the call, so nothing waits on the download. A body that fails
mid-stream ends the response early instead of raising an unhandled stream
error. xAI speech-to-text keeps one copy of the audio (the upload Blob)
instead of up to three. The voice changer and xAI STT hold credits through
withAiCreditHold.

* perf(ai-ocr): read Puter files only when the provider needs the bytes

resolveFileInput resolves and access-checks a Puter file and defers the read
to load(); loadFileInput is that plus the read. OCR resolves first: Textract
reads the object straight from S3 and downloads it only if Textract can't
reach the bucket, and only Mistral loads the bytes. A file over the size cap
is refused from its recorded size before any read. countPdfPages inflates
object streams on the libuv pool instead of the request thread.

* perf(ai): build image and video model catalogs once and share output saving

ModelCatalog holds what the public model routes serve (models, sorted names,
cost lines), computed once after boot instead of on every listing. The video
driver now awaits its model map before serving. resolveOutputPath and
saveGeneratedMediaToFS replace the copies of puter_output_path handling in
both drivers, using expandTildePath and normalizeAbsolutePath. The video
driver normalizes a copy of the caller's args instead of mutating them, and
its never-taken stream branch is gone.

* fix(drivers): abort the response when a streamed driver result fails mid-body

upstreamBodyStream now errors when the provider body fails instead of ending
cleanly, and DriverController pipes stream results through pipeStreamResult
(stream.pipeline): a failed source destroys the response so the client sees
an aborted transfer, the failure is logged once, and nothing is left to raise
an unhandled stream error. A client hanging up stops the source.

* fix(drivers): drop a streamed driver result when the caller already hung up

pipeline() throws synchronously on a closed destination, which surfaced as
a 500 alarm on every early disconnect and left the upstream body open.

* fix(ai): bound streamed upstream calls by time to first byte, not download time

upstreamFetch takes a `stream` option: the timeout covers the request until
the response (or a non-2xx error body) is read, then is cleared so a body
piped to the client drains at the client's pace. ElevenLabs TTS, xAI TTS and
the voice changer pass it; non-streamed calls keep the full-response bound.
A stalled upstream body is still cut by undici's body idle timeout.

* fix(drivers): type pipeStreamResult's response by the members it uses
2026-10-11 22:05:37 -07:00
Daniel Salazar 45f6e9d8ee refactor(drivers-wiring): AppService views and launch gates, batched app lookups, shared driver policy and guards (#4186)
* perf(workers): skip hot-reload lookups for files no worker is bound to

The fs.write.file / fs.remove.node / fs.move.node listeners listed every
workers.puter.* row of the file's owner on every event, for every user.
The event's FSEntry already carries the worker rows bound to it, so a file
with none returns before any I/O; a bound file reads just its own rows,
batched, from the primary.

* perf(apps): read the uid alias only after a direct lookup misses

Every AppDriver uid lookup fired an uncached system-KV GetItem for the
merge alias in parallel with the direct read, even when the direct read
hit. The alias is now read only on a miss.

* refactor(apps): one app view with the launch gates, built by AppService

AppDriver's serializer and its private-app / hosted-backing gates move to
AppService.views, built on a pure toAppView that is the one place
index_url is withheld. AppDriver, the subdomains driver's embedded apps and
the GUI shell view all use it. Visibility of protected apps
(filterReadable) moves with it.

Subdomain embeds (associated_app, app_owner) used to return index_url with
no gate; they now withhold it from a viewer who may not open a private app
or whose hosted backing is gone, and carry privateAccess, godmode and
feedback_enabled like a direct app read.

* refactor(apps): one toAppSummary; batch the recommended and taskbar lookups

SuggestedApps, RecommendedApps, /get-launch-apps and the taskbar each built
their own launch summary; they now share toAppSummary (the extra
feedback_enabled / external flags appear on the lists that lacked them).

getRecommendedApps awaited getByName once per recommended app (29) and
getTaskbarItems did the same per taskbar entry on every whoami; both now
make one batched call per key through the new AppStore.getByNames, which
keeps getByName's case-insensitive and old-name matching. The taskbar's
raw UPDATE of the user row goes through UserStore.update, and
/get-launch-apps checks hosted backings in one batch and reaches the
stores without casts.

* refactor(apps): app routes read through AppService; /delete-site calls the subdomains driver

AppController used the global driversContainers through a getter whose
boot-order comment no longer held; it uses the injected drivers and
AppService. GET /apps/:name looked each name up, read it again through the
driver and resolved private access a second time, emitting
app.privateAccess.resolveLaunch twice per private app; it now batches the
names and resolves once. POST /query/app did a lookup, a driver read and a
raw app_filetype_association SELECT per selector (up to 200, serially); it
now makes one batched lookup per key, one grant check and one filetype read.

The driver's read shape is a pure toAppReadView both use.
HostingController's /delete-site re-implemented the subdomains driver's
delete (protected check, delete, subdomain.delete event); it keeps its
owner-scoped 404 and calls the driver. Hosting and Desktop controllers drop
constructors that discarded the drivers argument, and trivial getters.

* refactor(drivers): one per-method policy validator and resolver; drop @Driver

Four policy blocks (rateLimit, concurrent, requireSubscription,
requireReputation) share the { default, methods } shape but each had its
own validator and resolver. They become validatePerMethod(value, label,
policy) and resolvePerMethod(cfg, method); error messages are unchanged.

The @Driver class decorator and the prototype keys it wrote had no users
outside its own tests; drivers declare their fields, and resolveDriverMeta
reads only those. DriverController takes the driver name from the meta it
already resolved instead of probing the prototype.
resolveDriverMethodRateLimit/Concurrent stay as deprecated aliases for the
email extension.

* refactor(drivers): shared actor guards and crud-q concurrency policy

The apps, subdomains, workers and notifications drivers each carried a
private #requireActor, two carried the same #requireVerified, and apps and
subdomains the same owner/app_owner write predicate. These become
requireContextActor, requireContextUserActor, assertActorEmailVerified and
actorOwnsRow in core/actor.ts; each driver keeps its own error for the
cases that differed (system actors still read apps; notifications still
answer 403 for an actor without a user). The identical crud-q concurrency
block is one CRUD_CONCURRENT constant.

The metering, installedApps and appTelemetry extensions re-checked an actor
their routes already require; those checks and the tests for them go.

* refactor(extensions): store methods instead of raw SQL in OSS extensions

installedApps and appTelemetry queried user_to_app_permissions directly
and metering read apps by name with its own SELECT. PermissionStore gains
listAppsGrantedByUser, listAppAuthenticatedUsers (with the email-share
check folded in) and countAppUsers; metering uses AppStore.getByName.

* refactor(server): one layer builder and one lifecycle hook runner

PuterServer built each of its five layers with its own copy of the same
loop (built-ins, then extension additions, into the layer and its export
container), and ran each lifecycle hook with five near-identical loops per
hook. Both are now one helper each: #instantiateLayer and
#runHook(order, hook). Construction order, the args each layer receives,
and hook order (start and prepare bottom-up, shutdown top-down) are
unchanged.

* fix(workers): tear down a deleted source's worker after its FK nulls root_dir_id
2026-10-11 22:04:23 -07:00
Daniel Salazar d162549c1d refactor(controllers): one signup, login, confirm and window.user path; userProtected/internalAuth route options (#4189)
* refactor(auth): one signup path for form, temp, OIDC and team accounts

Signup steps (validate hook, phone/card forcing, bonus verdict, provisioning,
signup events) move from AuthController and OIDCService into SignupService.
Username rules, availability and generation move to util/username.ts, and
user-row + group + default-folder creation to provisionUser, which
TeamService and DefaultUserService now share.

Behaviour fixes: a provisioned team seat joins default_user_group like every
other account, and generated temp usernames skip names whose home path is
occupied.

* refactor(auth): one web-session start for password and OIDC sign-in

startWebSession creates the session with the request's IP, user agent, origin
and host and sets the session cookie. OIDC sign-ins used to create their
session without that metadata, so they showed up blank in the session list.

* refactor(auth): one confirmEmail for the code and link confirmation routes

Both routes now re-run the email checks, refuse an address another account
confirmed, strip it from other rows, promote the group and push
user.email_confirmed to the account's sockets (with the originating socket id
on the code route). The link route's full-table UPDATE clearing other
accounts' pending change-email requests is dropped: /change_email/confirm
already refuses an address a confirmed account owns.

* refactor(auth): build window.user once for whoami and sign-in

buildUserDetails and scrubSensitive move from the whoami extension into
util/userDetails.ts. /whoami calls them; sign-in responses now return the same
user-actor payload, scrubbed, so the login copy no longer ships the phone
number and a no-reload login gets oidc_only/otp/directories up front. Taskbar
resolution is best-effort on both.

* refactor(http): userProtected route option replaces AuthController's route walker copy

The user-protected gate is materialized from a userProtected RouteOption
(implies requireUserActor), so AuthController no longer re-implements the
decorator walker to wire it per instance; those routes and the session
routes are plain decorators now. AuthRequired also counts requireVerified
and userProtected.

* refactor(http): internalAuth route option and a shared bodyRecord

EventsController and PeerController each checked x-puter-internal-auth by
hand with different compares; an internalAuth RouteOption now picks the
secret from config and one constant-time gate checks it. ShareController and
EventsController share bodyRecord instead of private copies.

* fix(fs): cap legacy /delete at 500 paths per request

The bulk form resolved, ACL-checked and removed an unbounded paths[] list one
by one. It now refuses more than FS_BATCH_WRITE_MAX_ITEMS (500), the cap the
batch write routes already use, before touching any entry.

* refactor(controllers): user and subdomain writes and reads through their stores

AuthController's 2FA setup/enable/disable and recovery-code writes go through
UserStore.update instead of raw UPDATEs plus a manual invalidate, and
/readdir-subdomains pages SubdomainStore.listByUserId instead of an unbounded
SELECT; the response shape is unchanged.

* refactor(auth): note SignupService's call-time peers

* fix(auth): type userDetails' taskbar deps independently of TaskbarDeps.clients
2026-10-11 22:00:53 -07:00
Daniel Salazar 576e893a5b refactor(services-core): read-only socket reauth, one token-grant read per check, shared grant/revoke and hosted-asset paths (#4187)
* perf(socket): re-check live sockets without touching their sessions

The 5-minute sweep reused the request-path authenticate(), which slid
every connected session's expiry and wrote user activity each tick, and
could fill SessionStore's touch-throttle map. authenticate() now takes
touch: false for re-checks; the sweep uses it, checks credentials
through runWithConcurrencyLimitSettled, and a call while a sweep runs
joins it instead of starting another.

* perf(acl): read a scoped token's grants once per check

ACLService.check did a Redis GET + JSON.parse of the token's grant list
for every ancestor x mode, and the permission scan did the same per
option. Both now read the list once (PermissionStore.listAccessTokenPerms,
which replaces the per-permission hasAccessTokenPerm) and test it in
memory.

* refactor(metering): incrementUsage goes through batchIncrementUsages

incrementUsage was a copy of the batch path that let an infinite amount
through and skipped NaN, where the batch path counts either as one unit.
It now calls batchIncrementUsages with a one-entry batch, so both apply
the batch clamping (documented on batchIncrementUsages). The overuse
alarm loses the single-path fields nothing sets any more.

* perf(metering): memoize an actor's spent addons on the increment path

Every increment and credit refresh read the actor's addons record from
the KV store. A record with no purchased credit left can only change
through a grant, and every grant announces credits-changed, so it is now
kept per actor beside the credit memo and dropped with it. Records with
credit left are still read fresh, since increments on other nodes draw
them down without announcing it. A read that straddles a drop isn't
kept.

* refactor(services): route raw table writes through their stores

- access_token_permissions: AuthService inserted grants one row at a
  time through a cast to the store's db client, read the authorizer, and
  deleted the manifest with raw SQL, then asked PermissionStore to drop
  its cache. PermissionStore now owns insertAccessTokenPerms (chunked
  multi-row insert), deleteAccessTokenPerms and
  getAccessTokenAuthorizerId, and invalidates inside the writes.
- TeamService's 2FA reset clears the OTP columns through UserStore.update.
- AppIconService's data-URL icon swap is AppStore.replaceDataUrlIcon.

* fix(user): correct cascadeDelete's FK note; seat cleanup is best effort

The header said most user_id foreign keys are ON DELETE SET NULL; the
migrations cascade nearly all of them and keep SET NULL for audit and
analytics history. forgetSeat ran unguarded after the row was already
gone, so a cache failure there failed the whole delete; it is now
logged and stepped over like the other teardown steps.

* refactor(acl): one grant/revoke path per holder kind

ACLService's user and group stat/set pairs were copies differing only in
which PermissionService methods they called. They now share #stat and
#set over a holder adapter ({ lockKey, query, grant, revoke }); lock keys
and errors are unchanged.

PermissionService's eleven grant/revoke methods repeated the same guard
blocks. Each keeps its own order of checks, now through shared helpers
(#requireUserId, #refuseAppActor, #assertCanManage, #requireUser,
#requireApp, #assertFitsColumn, #audit), so every refusal keeps its
status, legacyCode and wording. Audit writes stay fire-and-forget.

* refactor(auth): one hosted-asset token API keyed by kind

The private-asset and public hosted-actor cookies had separate create/
verify methods, four cookie-name getters and two identical options
getters, differing only in the kind claim and what the host field was
called. They are now createHostedAssetToken(kind, claims),
verifyHostedAssetToken(kind, token, expected),
getHostedAssetCookieOptions(), and the exported HOSTED_ASSET_COOKIES
names. Token claims, cookie names and every error message are unchanged;
puterSite and privateAppGate call the new methods.

* refactor(auth): one revoked/expired check for every actor builder

The session, app-under-user and access-token builders (and the access
token's parent-session check) each spelled out the same revoked/expired
tests. They now share #endedSession, which returns the same reauth
results; only a session token passes its auth_id. Adds coverage for an
access token whose parent session expires, is revoked, or is gone.
2026-10-11 21:55:20 -07:00
Daniel Salazar ecef154630 refactor(services): page fs subtree walks, gate share fan-out, one fs ACL helper (#4190)
* refactor(acl): read ancestor chains from FSEntryStore and share one fs ACL helper

getAncestorChain (split a path, batch-read the entries) moves to
FSEntryStore, with getAncestors for callers that want the rows. Services
read it from the store instead of calling FSService sideways;
FSService.getAncestorChain stays for drivers and extensions.

ACLService gains fsDescriptor (memoized chain), assertFsAccess,
assertFsCreate and refusal, plus the pure aclRefusal mapping. The eight
hand-rolled descriptor and safe-error copies in FSService, ShareService,
events/authorization, FSController, legacyFsHelpers, LegacyFSController,
AuthController and WebDAVController use them. Each route keeps its
status, legacyCode and message: fs routes answer access_denied on a 403,
the legacy routes still hide refusals from apps as 404, share and events
keep forbidden and their own 404 text.

* perf(fs): page recursive deletes and batch their event dispatch

A recursive delete read every descendant into memory, and an empty-check
for a non-recursive one did the same. It now walks the subtree a page at a
time in descending path order (children before the directory holding
them, so the parent_id cascade never takes a row before it is handled),
deleting objects and rows and publishing each page as it goes. A
non-recursive delete probes with one LIMIT 1 query.

Removals publish once per entry, after the row is gone (the same change
as the open fix for the double emit), with one surviving-ancestor chain
read at most once for the whole subtree. Events-API dispatch for a batch
runs as one background task at bounded concurrency instead of a
fire-and-forget promise per entry.

FSEntryStore gains hasDescendants and listDescendantsInOrder (keyset
paged by path, then id).

* perf(fs): copy directories a level at a time

A directory copy walked every descendant in memory and copied them one
by one: an insert per directory and, per file, an S3 CopyObject, a
batchCreateEntries of one and its read-back. It now pages the source in
ascending path order (parents before children) and takes each page a
depth at a time: rows for directories, links and empty files at bounded
concurrency, then the depth's S3 copies at bounded concurrency and one
batchCreateEntries for all of them. A single-file copy goes through the
same path.

A failed S3 copy still fails the call (a ghost file still answers 404),
but only after the objects that did copy have their rows, so nothing is
left in the bucket unreferenced. Copied entries publish per depth in one
dispatch.

* refactor(share): build masked share paths in one place

sharePathMask exports maskedRoot(owner, uuid, name) and
maskUnder(root, realPath). ShareService's maskedPathVia goes, its
maskedSelfPath and shareDeepLink's maskedSharePath build on them, and the
masker, maskUnderAnchor and learnShareRoots use maskedRoot. Output is
byte-for-byte the same; tests pin it.

* fix(share): send old_path on a recipient's item.moved

The desktop's item.moved handler reads old_path; ShareService sent the
former masked path only as from_path. Both are sent now.

* perf(share): skip the audience lookup for owners with nothing shared

Every fs write, create, rename, move and delete looked for share
recipients: an ancestor walk plus two share queries, for an owner who in
nearly every case has shared nothing. #sharesReaching now reads the
chain through FSEntryStore.getAncestors (no third copy of the walk) and
asks a per-owner "has anything shared" flag before the share queries.

The flag is the per-user cache events already keeps for subscriptions
(SubscriptionCache): one ShareStore.ownerHasShares read per owner per
TTL, coalesced across a burst. Creating a share (user, team, claimed
invite) or moving a subtree to a new owner bumps it locally and over
outer.pubsub.share.ownerShared; the TTL covers a process that misses
the bump. A failed lookup reads as shared, so the full lookup decides.

* perf(share): batch the per-row reads behind share listings

Share listings looked things up one row at a time:

- teams by id: TeamStore.getByIdsIncludingDeleted (the single-id read
  calls it) for listSharedByMe, listSharesOf and #revokeGroupSharesBy
- team shares per node and ancestor: ShareStore.listGroupOnFsentries,
  one query for the chain, in the same node-then-ancestor order
- #liveGroupGrants: PermissionStore.queryIssuerGroupPermsByPrefixes
  reads every issuer/group pair under every prefix in one query
- #revokeGroupSharesBy: the five canManagePermission checks per row run
  together instead of one after another
- ShareNotificationService #unblocked: users via getByIds and blocks via
  UserBlockStore.blockersAmong, two queries instead of two per member
- digest flush: entries are taken at bounded concurrency rather than one
  at a time

* perf(events): ask background consent once per holder and app on a revoke settle

Withdrawing events:background re-checked consent row by row, resolving the holder's actor and asking the permission service for every one of their background rows. Rows of one holder and app stand on one consent, so it is asked once and shared.

* refactor(events): drop the registry's unused push fields

notify, defaultDelivery, PushMessage, NotifyProjection and pushProjection were read only by registry.test. Every subject was unpushable and broadcast by default; the subscription's delivery class stays in DeliveryClass, which the stores use.

* refactor(share): retire trashed shares through onEntryOwnerChanged

onEntryTrashed was a copy of onEntryOwnerChanged: the same subtree share sweep, grant retire and row delete.

* fix(fs): page subtree walks by depth, not path

MySQL sorts a PAD SPACE string on its first max_sort_length (1024) bytes,
so under utf8mb4_unicode_ci paths past about 512 characters tie and fall
back to id. A file moved under a newer directory then sorted after it: the
delete walk reached the directory first and the parent_id cascade took the
file without its object delete or event, and the keyset skipped or
repeated rows. Pages now order by segment count, then id, which no
collation can reorder.

* fix(share): cache an owner as shared when the bump arrives

A bump only dropped the cached answer, so the next lookup could read a
replica that hadn't applied the new share yet and cache "nothing shared"
for another TTL. The bump now records "shared" outright.

* fix(fs): probe a directory for children by parent_id, from the primary

* fix(fs): keep the copied-size check in the batched directory copy

A copied file records the size CopyObject actually duplicated; growth past
the source row is charged against the allowance, and a copy that no longer
fits is deleted and refused, as a single-file copy already does.
2026-10-11 21:54:13 -07:00
Daniel Salazar c815dc82a1 refactor(core-util): shared bounded map, cheaper route lifecycle, DDB/DB/rate-limit/gate dedup (#4185)
* refactor(util): add BoundedTtlMap for core caches and log throttles

One size-capped, LRU-evicting map with an optional TTL and a once-per-window
shouldEmit helper. Replaces the hand-rolled maps in the DynamoDB throttle and
repair loggers (the repair one was unbounded), the Slack repeat throttle, the
auth probe's reauth log throttle, and the card-verification status memo.

* perf(http): skip route lifecycle work when nothing listens

The per-route lifecycle middleware built a payload, opened an
event.emitAndWait span and hooked finish/close on every request, listeners
or not. Each phase now checks hasListeners first. EventClient's three copies
of the key-prefix walk become one memoized #matchKeys.

* perf(ddb): write batch chunks concurrently and retry the TTL sweep

batchPut/batchDel sent their 25-item chunks one after another; they now run
up to four at a time through runWithConcurrencyLimit, and a failed chunk
stops the ones not yet started. batchGet and the batch write share one
#sendWithUnprocessedRetry instead of two copies of the backoff loop, and the
TTL sweep goes through #batchWrite, so the unprocessed deletes it used to
drop are retried. The AWS and local clients are built by one #bindClient.

* refactor(db): share migrations and tryHardRead in the base client

MySQL and Postgres each carried a copy of the migration-path runner and of
the replica-racing tryHardRead; both now live on AbstractDatabaseClient
(applyMigrationPaths, and tryHardRead gated on hasReadReplica), and SQLite
drops its own tryHardRead override. Removes MySQL reinitPrimary,
reinitReplica, isRetriableError and readWithRetry and the base requireRead,
which had no callers outside tests. SQLite prepared statements and the
Postgres SQL rewrite are memoized per query text in a bounded map.

* refactor(rate-limit): route every entry point through one charge()

The nine rate/peek/acquire entry points each repeated key and limit
resolution, the backend call and the fail-open catch. They now share
charge(), routeKey() and guardSlot(); log lines are unchanged. Driver
concurrency slots release at most once like the other slot handles, and
checkRateLimit takes { failOpen: false } for limits that must hold when the
backend is down.

* refactor(util): add secretsEqual for constant-time secret comparison

One HMAC-then-timingSafeEqual helper, safe for inputs of any length, so
callers stop hand-rolling the length guard. File signature verification
uses it.

* refactor(fs): infer signed-file MIME types with mime-types

mimeFromName looks extensions up in mime-types instead of a 35-entry inline
table, keeping the two types the table reported differently (ico, wav). The
purpose-labelled signing key is derived once per secret instead of on every
signature, where signFile derived it twice.

* refactor(http): build route gates from one gate() helper

Each gate repeated a try/catch -> next(err) wrapper or a .then(next, next)
chain around an assertion. gate(check) does that once: a throw or rejection
goes to next(err), and a synchronous check still calls next synchronously.
The credits, subscription and reputation gate files held nothing but such a
wrapper; their factories move into gates.ts and the files go.

* perf(hosting): build the puter-site hosting config once

createPuterSiteMiddleware rebuilt the normalized hosting-domain config on
every hosted-site request; it is derived from static config, like the domain
lists the middleware already builds at construction.

* fix(db): tryHardRead's replica leg doesn't fail over to the primary

MySQL's read() retries on the primary after a load-shed or transient
replica error, so routing tryHardRead through it read the primary twice
on that path. The replica leg now reads the replica alone.

* fix(ddb): settle in-flight batch chunks before rethrowing

A failed chunk stopped unstarted chunks but rejected while others were
still retrying, so writes could land after the caller had handled the
error and invalidated its cache.
2026-10-11 21:53:33 -07:00
Daniel Salazar b5eefaa9cc fix(fs): include supplementary-character names in subtree path queries on MySQL (PUT-2135) (#4198)
* fix(fs): include supplementary-character names in subtree path queries on MySQL (PUT-2135)

MySQL ends the idx_fsentries_path range for `path LIKE 'p/%'` at 'p/' +
U+FFFF, and utf8mb4_unicode_ci sorts supplementary characters above it, so
a child whose name starts with one (most emoji) was skipped with its whole
subtree by listing, delete, copy, rename/move rewrites, counts and sizes.
On MySQL, FSEntryStore now looks up those children by parent_id and ORs a
LIKE pattern for each onto the subtree match, recursing into directories.
Every extra pattern starts with the prefix, so only the scanned ranges
widen; rewrites past 500 patterns run in several statements.

The prefix rewrite also passed a UTF-16 length to SUBSTR, which counts
characters, so a rename under a path holding an emoji dropped characters
from every descendant path on all engines. It now counts code points.

* test(fs): seed the supplementary-name subtree with createNonFileEntry
2026-10-11 21:21:18 -07:00
Daniel Salazar af9ebb0ef1 fix(ai): map BytePlus video and Cloudflare image upstream errors at the driver boundary (#4184) 2026-10-11 21:20:27 -07:00
Daniel Salazar 2131b980ce fix(redis): release locks with an atomic compare-and-delete (#4183)
* fix(redis): release locks with an atomic compare-and-delete

ACLService, TeamService and the share digest flush each hand-rolled a SET NX
lock whose release was a GET-then-DEL (or a blind DEL), so a holder whose TTL
lapsed could delete the next holder's lock. They now share withRedisLock,
which claims with a random token and releases via a Lua compare-and-delete.

* chore(team): place the redis lock import with the other runtime imports
2026-10-11 21:19:37 -07:00
Daniel Salazar 96909b0800 fix(fs): send the same outer.gui.item.* payload from every fs controller (#4179)
* fix(fs): send the same outer.gui.item.* payload from every fs controller

The v2 /fs/move and /fs/delete routes spread the raw camelCase FSEntry into
item.moved / item.removed and named the old path `from_path`, while the
desktop reads the legacy snake_case shape and `old_path`. One helper,
emitGuiItemEvent, now builds the payload for LegacyFSController,
FSController and WebDAVController.

v2 and WebDAV copy/move with overwrite now also emit item.removed for the
entry they replace, as the legacy routes do, and v2 /fs/move forwards
new_metadata to the service like the legacy /move.

* fix(fs): announce a replaced entry before the one replacing it

The desktop hides rows by path on item.removed, so a removal sent after the
overwriting copy's added (or move's moved) hid the new row too. The shared
emitter now takes the replaced entry and sends its removal first, awaited,
on the v2, legacy and WebDAV copy/move paths.
2026-10-11 21:18:47 -07:00
Daniel Salazar 906d6b7886 fix(email): one canonicalization and blocklist for every address path (#4176)
Share invites, share notifications and app feedback checked addresses through
EmailClient.clean/validate, a second copy of the rules in util/email.ts that
stripped dots on iCloud, stripped `+tags` on every domain, and matched the
blocklist case-sensitively. They now use util/email.ts: isBlockedEmail for the
blocklist, abuseKey for the per-address notification budget, and cleanEmail
for the invite digest key. EmailClient keeps transport and templates only.

isBlockedEmail also lowercases the configured suffixes, so a mixed-case
blocklist entry still matches.
2026-10-11 18:24:28 -07:00
Daniel Salazar 6a938bd8e8 fix(db): detect duplicate keys with isUniqueViolation everywhere (#4175)
FSService, OIDCStore, UserStore and TeamService each carried their own
duplicate-key check. FSService only matched mysql and the bare
SQLITE_CONSTRAINT code, so the mkdir/move race recovery never ran for a
SQLite or Postgres unique violation. OIDCStore matched any message
containing "UNIQUE" or the bare SQLITE_CONSTRAINT code, UserStore any
SQLITE_CONSTRAINT_* code, and TeamService any message mentioning unique or
duplicate. All four now use util/dbError's isUniqueViolation; UserStore
still checks the index name on top.
2026-10-11 18:24:13 -07:00
Daniel Salazar 847a4fc031 fix(fs): route thumbnail writes through FSEntryStore (#4174)
* fix(fs): route thumbnail writes through FSEntryStore

Add FSEntryStore.updateEntryThumbnailByUuid, which sets a thumbnail with
no owner check (callers authorize) and refreshes the entry cache, with an
optional expected-value guard.

The thumbnails extension's inline data: URL migration, its copy
duplication, and the legacy /update-fsentry-thumbnail endpoint now write
through it instead of raw UPDATEs, so cached entries stop serving the old
thumbnail. The migration validates the image like the upload path, awaits
the write, and only repoints a row that still holds the data URL; a read
that loses that race deletes its upload.

* fix(fs): report a thumbnail update by whether the row changed

updateEntryThumbnailByUuid returned the refreshed entry, which is null
when the cache misses and the replica hasn't seen the row yet. The
read-time migration took that null for a lost race and deleted the object
the row had just been pointed at. It now returns whether the UPDATE
changed a row; the cache refresh still runs after it.
2026-10-11 18:23:58 -07:00
Daniel Salazar 3296526e8e fix(ai-chat): derive Azure model pricing from the OpenAI and xAI catalogs (#4173)
Azure entries that front an OpenAI or xAI model now come from a `mirror`
helper that copies every non-identity field from the source catalog entry.
The hand-copied entries had drifted: none carried `long_context_pricing`, so
long prompts on Azure-served gpt-5.4 and Grok 4.3/4.20 were billed at 1x;
grok-4.3 had lost `pdf` input and the Grok 4.20 deployments still listed the
old 2M context. Models no longer in the direct catalogs stay inline.
2026-10-11 17:57:14 -07:00
Daniel Salazar c553a380ae fix(ai-chat): forward max_tokens to xAI; moderate Azure through OpenAI (#4172)
xAI sent a fixed max_tokens of 1000, ignoring both the caller's value and
the credit gate's cap. It now forwards max_tokens like the other
OpenAI-compatible providers and omits it when unset.

Azure chat's checkModeration threw "Method not implemented", so
`moderation: true` on an Azure route metered the completion, then failed it
as a route error: Azure was marked unhealthy and the fallback billed a
second completion. Azure's v1 API has no /moderations endpoint, so both
Azure providers now delegate moderation to the OpenAI chat provider
(Azure chat via its Responses sibling, as it does for web_search).

A moderation call that throws is now a 400 `moderation_unavailable` that
the driver treats like `moderation_flagged`: final, no fallback, and the
route stays healthy.
2026-10-11 17:55:38 -07:00
Daniel Salazar cf9df8719a fix(alarm): forget idle alarms and cap how many the client holds (#4171)
AlarmClient kept every alarm id it had seen for the life of the process,
and ids that embed a user, key or invoice grew it without bound. The
registry is now ordered by last occurrence: alarms idle for 24 hours are
dropped on the next create, and past 1000 the least recently raised goes
first. Clearing or evicting an alarm only removes its short-id alias when
the alias still points at it.
2026-10-11 17:54:26 -07:00
Daniel Salazar 7ad963bacf fix(cache): broadcast fs entry and team invalidations to peer regions (#4169)
* fix(cache): broadcast fs entry and team invalidations to peer regions

FSEntryStore's invalidations called publishCacheKeys without broadcast,
so a delete, move, rename or overwrite only cleared the writer's
cluster; other regions served the old row until the 60 s TTL. They now
broadcast. Batch paths (deleteEntries per chunk, the batch overwrite)
send one event for the whole batch. Read-path backfills stay local.

TeamStore's busts were a local redis.del, so a removed member kept
passing getMembership in other regions until the TTL. They now go
through publishCacheKeys with broadcast, and softDelete busts every key
in one call.

Remove FSEntryCacheInvalidationEventHandler. It re-derived the same
invalidations from outer.gui.item.* with a replica SELECT per holder,
and its fs.remove.node branch never ran (FSEntry has no get()). The
store's own broadcast covers what it did.

* fix(fs): cap how many entry invalidations a node broadcasts per window

FSEntryStore now broadcasts at most 10,000 entries' cache invalidations
per broadcast flush window (outbound_flush_ms, 2 s by default). Past
that, keys are dropped locally only and peers fall back to the 60 s
entry-cache TTL; the store logs once per window when this happens.

Each flush goes to a peer as one POST that is re-sent whole on failure,
so a large recursive delete could otherwise build a batch the peer
rejects on every retry. The budget is per store instance, not per call,
so a delete issued page by page is bounded too, and single-entry writes
count against the same budget.
2026-10-11 17:53:26 -07:00
Daniel Salazar 0066d49f74 fix(fs): emit fs.remove.node once per entry on recursive delete (#4167)
#removeDescendantsStorage emitted fs.remove.node for every file
descendant while grouping S3 keys, and remove() emitted it again for
every descendant after deleting the rows, so file listeners ran twice.
Drop the storage-pass emit; remove() now emits once per entry after its
row is deleted.
2026-10-11 17:52:27 -07:00
Daniel Salazar b67acdbf6f fix(stores): return written rows without reading them from a lagging replica (#4166)
* fix(stores): read rows back from the primary after writing them

ShareStore (create, upsertPending, upsertActive, upsertActiveGroup,
applyPending, upsertAnyone), AppStore (create, createFromOrigin, update,
claimOwnership), NotificationStore.create, SubdomainStore.update and
NotificationDriver.create returned the row they had just written by
reading it back through `db.read`, which goes to the replica. Under
replica lag that came back null or stale. Those read-backs now go to the
primary, through a `primary` option on the getters. AppStore caches the
primary copy on create, and claimOwnership no longer reads through
getById, which cached the replica's pre-claim row on a miss.

* fix(stores): skip the notify read-back; read OIDC link conflicts from the primary

NotificationStore gains `insert`, which writes and returns the uid;
`create` is now `insert` plus the primary read-back. NotificationService
.notify ignores the row, so it uses `insert` and no longer pays a primary
round trip per recipient. NotificationDriver.create still reads the row
back from the primary.

OIDCStore.link reads the conflicting row from the primary after a unique
violation, so a replica that has not seen another account's link can no
longer make the call report success.

* perf(stores): return written rows instead of reading them back

ShareStore.create, NotificationStore.create, AppStore.create and
AppStore.createFromOrigin build the row they return from the values they
inserted and the insert id, instead of reading it back from the primary.
The DB-defaulted creation timestamp is now written explicitly from unix
seconds (`timestampFromUnixSql`), so the returned value matches what a
read returns (`timestampValue`); postgres INSERTs add RETURNING id. The
AppStore rows fill every column the INSERT leaves to the schema and are
cached as before.

ShareStore.upsertPending's re-invite branch selects the full pending row
in its dedupe read and returns it with the new mode and data; it falls
back to the primary only when the UPDATE matched nothing.

The remaining primary read-backs stay and say why: the share upserts and
the new-invite insert (an upsert's surviving uid/created_at and the
entry-owner subquery are only known to the primary), applyPending and
claimOwnership (only an id or uid in hand), AppStore.update and
SubdomainStore.update (the base row is a replica copy that would be
cached), NotificationDriver.create (notify returns only the uid) and the
OIDC link conflict.
2026-10-11 17:51:38 -07:00
404oops 64db150a90 fix(ai): bound streaming buffers and normalize input errors 2026-10-11 21:14:16 +02:00
Daniel Salazar f88cadb4b9 fix: a copy records the size of the bytes it duplicated (#4197)
A copied file's row took its size from the source row the request had
resolved, which can predate an overwrite whose bytes CopyObject then
copied. The copy now reads the new object's size and records that. When
it is larger than the source row said, the allowance is checked again
against the real size; a copy that no longer fits is refused with 413
and its object deleted.

If the size check fails, the copy falls back to the source row's size,
as completing a signed upload already does.
2026-10-11 10:48:52 -07:00
Daniel Salazar aef0a32d72 fix(ai): hold credits for image and video generation (#4164)
Image and video providers checked the balance with hasEnoughCredits /
getRemainingUsage and took no hold, so concurrent requests each saw the
full balance until the first one recorded usage. They now reserve the
estimate through the cost-factored reserveAiCredits and release it in a
finally block once usage is recorded, like chat, speech and OCR.

The video duration clamp now prices seconds at the model's cost factor
and holds the clamped clip's cost for the job (poll window plus create
time). Replicate holds the output-only estimate before fetching inputs
and the input surcharge on top once it is measured.
2026-10-11 09:18:13 -07:00
404oops 800e300a8c fix(ai-tts): resolve ElevenLabs voice names before synthesis 2026-10-11 16:33:05 +02:00
Daniel Salazar 491729e060 fix: backend test flakes and a socket shutdown hang (#4170)
- SocketService's prepare-shutdown hook awaited io.close(), which also
  closes the http server and waits for every HTTP connection to end. A
  connection mid-request when close() runs stays open for the keep-alive
  timeout, so shutdown stalled before PuterServer severs connections. The
  hook no longer waits on the http close.
- Test servers disable Ollama unless a test configures it. Its boot probe
  logged after files had ended, which failed shared-worker shards with
  EnvironmentTeardownError.
- vitest ignores that teardown error for a console call still in flight.
- setupPuterTestEnv boots again on a fresh port after EADDRINUSE, and a
  failed setupTestServer start shuts the server down.
- UploadReservationStore expiry tests pin the clock instead of racing a
  50 ms TTL.
2026-10-10 19:34:32 -04:00
Daniel Salazar d4589caefe fix: session store revokes (#4168) 2026-10-10 07:55:43 -07:00
Daniel Salazar 4fca4809b9 fix(health): discount total event-loop stall time from db liveness latency (#4160)
The stall watcher kept only the longest gap, so a saturated loop with many sub-second stalls still failed database-liveness. It now sums blocked time across the probe (ignoring timer jitter) and reports total and longest.
2026-10-09 18:14:48 -07:00
Daniel Salazar cc8b9858a2 feat: let extensions claim a subdomain (PUT-2100) (#4155)
`extension.claimSubdomain(name, { authorize, checkRoute? })` lets an
extension serve `<name>.<domain>` without a hard-coded reserved name.

- A claimed host is dispatched where the unknown-subdomain 404 used to be,
  before CORS, the OPTIONS responder, body parsing, auth and every route
  (including `subdomain: '*'` ones).
- `authorize` returning anything but true, or throwing a non-HttpError,
  yields exactly the unknown-subdomain 404. A thrown HttpError goes to the
  error handler.
- Routes whose `subdomain` is the claimed name (decorated controllers and
  extension routes alike) are materialized onto the claim's own router, so
  they only run after `authorize` and never on the main app. Their options
  are limited to ones that don't rely on the main pipeline's actor; an
  optional `checkRoute` lets the claim refuse routes at boot. Case-variant
  labels and claimed names mixed into subdomain arrays fail the boot.
- The unknown-subdomain check now compares the port-stripped hostname and
  requires a dot boundary, so a host with an explicit port behaves like one
  without it. Hosts served by the local worker proxy still pass through.
2026-10-09 18:14:04 -07:00
Daniel Salazar 6060de8fbd fix(alerts): overspend alarm in credits at warning; no-choices completions stop paging (#4154)
- The per-account overspend alarm now reads in credits and raises at
  warning so it reaches the pager instead of only chat.
- A non-stream completion with no choices (an upstream answering 200 with
  an error body) threw a TypeError that classified as internal_error and
  paged critical. It now throws bad_response, which falls back like any
  other empty response and surfaces as a 400.
2026-10-09 18:13:44 -07:00
Daniel Salazar 0e0cedb02b fix(auth): no email password recovery for built-in admin accounts (#4153)
- /send-pass-recovery-email for `admin`/`system` (by username or email)
  returns the usual generic message, writes no token, sends no mail, and
  raises a critical alarm with the requester's IP, user agent and, when
  signed in, their user.
- /verify-pass-recovery-token and /set-pass-using-token reject recovery
  tokens for those accounts as no longer valid.
2026-10-09 18:13:22 -07:00
Neal Shah 43094a84a0 PUT-1901 (#4159) 2026-10-09 20:12:45 -04:00
Neal Shah 5fb025c109 Misc subdomain and worker fixes (#4157)
* PUT 1902

* LocalWorker fetch policy fixes for 127.0.0.1
2026-10-09 19:59:52 -04:00
Juan Fernando Castro 50bcdc9c6c fix: let a revocation be seen by the reads and the nodes that outlive it (#4151)
* fix: let a revocation be seen by the reads and the nodes that outlive it

Four places where a revoke lands on the primary and the next reader can
still be told the old answer.

The u2a cache key was dropped but not re-filled, so the next miss read a
replica that may still carry the row and cached it for the TTL -- and a
revoke's own settling pass is usually that miss. It re-fills from the
primary, as the u2u side already does.

A revoked session dropped its keys the same way. It now leaves the
revoked row in the cache, read from the primary, so there is nothing for
a lagging replica to re-introduce.

A link share is its own authority rather than an index over a
permission, so a replica still carrying a withdrawn row answers with it.
For a short window after one is withdrawn, those reads go to the
primary.

Socket eviction rode a local event, so it only reached the node that did
the revoking. It is announced on the fan-out channel too, which is where
the node actually holding the connection hears it.

* refactor: carry the revocation on outer.* instead of outer.pubsub.*

Eviction runs through `io.in(room).disconnectSockets()`, which goes via the
socket.io Redis adapter a cluster shares, so one node handling the event
disconnects that user on every node in it. Fanning out to siblings had each
of them repeat the same cluster-wide disconnect.

The adapter's Redis is per-cluster, so peer clusters still need the hop --
`outer.*` reaches them and skips the sibling fan-out. The local
`auth.sessions.revoked` goes with it: `outer.*` fires on the emitting node
too, and nothing else subscribed.
2026-10-09 16:36:26 -04:00
Juan Fernando Castro ff35fbd66e Share attribution and cascade: finish PUT-1856 and PUT-1857 (#4119)
Replayed onto main after #4118 squash-merged. Same tree as the reviewed
head (8d8bd871f); only the redundant merges from the old base are gone.
2026-10-09 16:23:18 -04:00
Juan Fernando Castro 1ff5d93244 fix: a signed overwrite lands on the live key only once its completion is accepted (#4136)
* fix: a signed overwrite lands on the live key only once its completion is accepted

A presigned overwrite URL was signed against the live object key, so the
browser's PUT replaced the owner's bytes directly. That PUT never passes
through Puter, which leaves the access check on `completeWrite` with
nothing left to protect: by the time it runs, what /read serves has
already changed.

The upload now goes to a key of its own. `completeWrite` is already the
authorized step, so that is where the staged object is copied onto the
live key, server-side, and the staged one dropped. The entry keeps its
uuid, so everything written against it still holds. Aborting deletes the
staged object, which the old shared key made unsafe to do.

`copyObjectOfAnySize` carries the copy past the single-request ceiling
by copying part by part, so a large file is no different.

Two behaviours follow from the key no longer being shared: completing an
overwrite that uploaded nothing is refused rather than reported as an
overwrite of unchanged bytes, and abort always drops the session's
object. Only the two presigned paths changed; `prepareBatchWrites`
uploads server-side within one authorized request and still writes in
place.

* fix: the review's findings on the staged overwrite

The batch signed path still signed against the live key, so the route
the SDK's signed upload actually takes was unchanged. It stages now,
which also makes the abort simplification safe: the guard that had
protected a shared key was removed while that path still shared one, so
aborting a batch overwrite would have deleted the owner's object.

The completion is allowed against a path but lands on the uuid captured
at start, and a move between the two puts the bytes somewhere the caller
was never authorized for. The promote refuses unless the entry is still
where the check looked.

The batch promote failing left every session pending with its lease
held; it now fails them as the rest of the function does.

The pending GUI tile and the thumbnail key were keyed on the staged
uuid, which no entry ever carries, so neither correlated with what the
completion emitted. Responses carry `entryUid` for that.

* fix: do not treat a pre-staging session's key as this upload's own

A session signed by the previous release points straight at the live key,
since `objectKey` was `existingEntry?.uuid ?? uuidv4()`. Completing one is
already handled -- the promote returns early when the two keys match -- but
abort deleted `objectKey` unconditionally, which during a rolling deploy
would take the owner's file rather than the upload's litter.

Abort now deletes only a key the upload owns. The multipart branch was
always safe: discarding parts never touches an object already in place.
2026-10-09 16:22:36 -04:00
Juan Fernando Castro 86cc3490f7 fix: settle which plan an org seat resolves to, and confirm a seat's address (#4139)
* fix: settle which default plan an org seat resolves to, and what counts as paid

Default subscription resolvers ran in registration order, which is boot
order: whichever layer registered first answered first. A seat is
provisioned by a team and its address is never confirmed, so the seat
resolver now states a priority and wins over anything a deployment
infers from an address.

The seat cap asked whether the owner's plan was outside the free set,
which a default plan can be without anyone having bought it. Metering
now reports whether an explicit subscription answered, and the cap rests
on that.

* fix: the review's findings on plan resolution

`fromDefault` was read off which resolver answered, not off which policy
applied. An explicit id naming a policy nobody registered falls through
to the default one, and that read as bought. It follows the applied
policy now.

A deployment with unlimited metering has no explicit resolver, so every
owner on it became unpaid and its teams dropped from the paid seat cap
to the free one. Unlimited answers for what a paid plan would, so it
counts as one.

Also: the new constants had been placed between `FREE_SUBSCRIPTION_IDS`
and its docblock, the register docstring still described first-wins, and
prettier had reflowed an untouched licence header.

* feat: a seat confirms the address its team typed in

An address on a seat is the team's claim about someone else's inbox, and
nothing asked the inbox. It is now demanded and sent a code, so the
address counts for nothing until the seat hands it back. Resolvers that
read a domain are the reason this matters.

No address is still fine and demands nothing: the team is the trust
anchor for the account, not for a mailbox it does not hold.

`/confirm-email` allows an unconfirmed actor, so a seat can sign in with
its temporary credential and clear the demand; nothing locks it out.

* fix: clear the confirmation demand with the address it names

`clearUnconfirmedEmail` releases a seat's address so the inbox's owner can
sign up with it. The demand to confirm that address has to go at the same
write: the gate reads `requires_email_confirmation && !email_confirmed`, and
once the address is gone there is nothing left to send a code to.

* fix: keep a seat's address correctable, and only ask what can be answered

Review of the confirmation demand turned up two ways a seat could be left
holding one it had no way to meet:

- With no mail transport configured, the code is never sent and a resend
  cannot reach the seat either, so the demand stood with nothing able to
  lift it. The seat is now gated only when a code can actually travel.
- An address the owner mistyped gated the seat on an inbox nobody could
  read, and the gate itself closes `/user-protected/change-email`, the
  route the seat would have fixed it by. `updateSeatEmail` lets the owner
  who typed the address retarget it, and refuses once it is confirmed,
  when it belongs to whoever proved it.

Also resolves the owner's plan sequentially, so the second read takes the
cache the first wrote and the pair cannot come from two resolutions, and
restores the licence header a jsdoc reflow had rewrapped.

* fix: let the confirm-email window open before the profile has loaded

Booted from a stored token, the window renders before `window.user` is
populated, and reading `.email` off null threw inside the window itself --
so the one screen that could have asked for the code never drew, leaving a
blank page. Reproducible without a team: sign up, then load the GUI again
before confirming.

The address is now encoded rather than interpolated raw, and both forms of
the line are i18n keys; it was hardcoded English before.
2026-10-09 16:17:51 -04:00
Juan Fernando Castro 7de8360a6e Sharing cleanup: bounded outbound listings, and what see does not reach (#4118)
* fix: keep a file name to one line in a subject, and the inbound total to the account

**A name reaches the digest subject as the user typed it.** Subjects compile
with `noEscape` on purpose -- a header is not HTML, and escaping would show
entities to the reader -- and the comment there says free-form values collapse
whitespace upstream. A file name does not: `#assertUsableName` refuses only an
empty name, a slash, `.` and `..`, so a name may carry newlines. It is
collapsed to one line and bounded before it goes in.

**`includeTotal` answered an app with the account's figure.** The page is
filtered by what the caller can reach -- an app sees only what its user handed
it -- while `countByHolder` counts every inbound share. The docs already call
`total` an approximation for the filtering; they did not cover an app reading
the size of everything else. It is answered only for a caller acting as the
account now, and the page documents that.

The team-share fixture built its actor as `{ user }` with no `effectiveApp`,
which reads as an app rather than a session. A session acts as the account, so
it says so.

* fix: stop `see` reaching inside a folder, and bound mode changes

**A `see`-only folder recipient was pushed every child's name, size and type.**
The fan-out picked holders by reach -- any live grant on a reaching root -- and
`see` is a grant. But `see` says a node is there; `list` is what says what is
inside it, and the payload carries exactly what `list` would: name, path,
`is_dir`, type, size, modified. Descendants now go only to a holder whose mode
reaches inside. A change to the shared node itself still goes to everyone who
holds it, which is what `see` covers.

**Mode changes skipped the daily quota entirely.** Not spending share budget is
right -- changing a mode is not new reach, and the existing test says so -- but
unbounded it lets one account write a pair as fast as it can ask. They have
their own budget now, ten times the share limit by default and configurable,
so the exemption stays an exemption from *that* budget rather than from any.

* fix: let a half-finished downgrade be finished, and stop Trash answering

**A mode downgrade that failed between its two halves could not be repaired.**
`setUserUser` grants the new mode and then revokes the superseded ones. If the
revokes did not all land, the higher grant stood -- and the retry returned
early, because the mode it was asked for was now present. "Nothing to do" now
means this mode is the only one standing, so asking again is what finishes it.
The same two halves, and the same early return, were in the group variant.

**Trashing someone else's item answered for what is in their Trash.** Moving
into the owner's Trash is deliberately authorized without write on it -- that
is what lets a recipient delete from a shared folder -- but a name collision
there came back as `An entry already exists at /owner/Trash/<name>`. The mover
picks the name, so that is a question they can ask about a folder they cannot
list. It dedupes instead, which is also what the GUI wants: trashing should
not fail over a name.

* perf: record the entry's owner on the share row

The outbound listings answer "shares on things I own that someone else issued"
by joining `share` to `fsentries` and filtering on the owner there. Neither side
of that join bounds the work: driven from `fsentries` it walks every file the
user owns, driven from `share` it walks every share until it has a page. For a
user who owns a lot and delegates nothing, that is the whole table for an empty
result. `#outboundAppsSql` is worse than the ticket says -- it has no LIMIT at
all, so every call aggregates the join across both tables.

The owner now rides on the row, indexed with `id`, so all three read it
directly. An entry cannot change owner -- a move out of the owner's tree is
refused -- so it is written once, with the row.

Set in SQL from `fsentry_id` rather than threaded through callers: four insert
sites, each of which already has the id, and none of which can now forget.
The risk in denormalising is a write path that does: a row with no owner
recorded is a share that silently stops appearing in its owner's listing. The
new test covers exactly that -- it fails when one insert is made to write NULL.

Verified against a real boot: a pre-existing share row, on a database that
genuinely predates the column, comes out with the owner its entry has.

* fix: the review's findings on this branch

**The Trash collision still had both halves open.** `overwrite` was checked
before the trash case, so a caller setting it -- it is wire-supplied -- could
still delete whatever it collided with in the owner's Trash, and the 409
survived for anyone who did not. The trash case is decided first now.

Dedupe alone did not close it either: a name that changes only when there is a
collision answers the same question the refusal did, and the mover sees that
name even though the path is masked. Trashing someone else's item renames it
every time, so the result carries no signal. `original_name` already records
the real name, which is what a restore reads. Two tests pinned the old name;
they say what it is now.

**The owner column had no fallback.** Reads went only through it, so a row
written by anything that predates it -- a pod mid-roll -- would never appear in
its owner's listing again. The three reads now also cover `IS NULL` rows
through the old join, indexed on the null and empty once none remain.

**The grant became conditional on a replica read.** `if (!already)` skipped it
on a stale row while the revokes below stayed unconditional, which could leave
the holder with nothing where the old early return left their access alone. It
grants unconditionally again; only the early return changed.

**The re-issue budget covered one path of four.** Team, link and invite
re-issues reserved nothing, so three ways round it were left open. They share
one `#reserveRemodeQuota` now.

Two docblocks had been pushed off the functions they describe, and two comments
still described the join that is gone.

* chore: one-line the comments added here

* test: measure the prefix walk by what it keys on, not by the clock

`checks many long paths without re-reading every prefix of each` gave
the walk 1500ms of wall clock. It measured nothing useful: the shape it
guards against — keying each step on the accumulated prefix text rather
than on the parent id — costs 29ms against the current 8ms on that
input, so both ends sit far under the bound. Under coverage the same
test ran 1511ms and failed, which is the only thing the number could
ever report on.

`PathPrefixIds` now counts the characters it keys on, and the walk is
asserted on that: proportional to the input, and twice the depth costs
twice as much rather than four times. Keying on the accumulated text
fails both. The original case keeps its real assertion, that a request
this size is refused.

* fix: keep the recorded entry owner current, and the trash guard on the retry

A move into someone else's tree re-owns the whole subtree, which the
migration comment claimed could not happen. The shares on it are
re-pointed at the new owner, or they drop out of the listing the column
keys and stay in the old owner's.

The unique-violation retry re-entered the collision resolver without the
trash flag, which is the one call site the guard was not threaded
through.
2026-10-09 16:14:23 -04:00
Daniel Salazar be78281d9d ci: shard backend tests and share workers across server-booting files (#4149)
The suite was CPU-bound on one runner, and about half of that CPU went to
re-importing the backend for every test file.

- Run the suite as 4 shards and merge their blob reports for coverage.
  `test (pr)` stays as the required check and fails if any shard fails.
- Drop the per-PR base leg; pushes to main publish the coverage PRs
  compare against.
- Files that boot a server without module mocks or extensions run in a
  `shared-worker` vitest project (`isolate: false`), so the backend is
  imported once per worker. testSharedWorkerSetup.ts resets the redis
  mock, configContainer and extensionStore between files.
2026-10-09 01:50:33 -07:00
Daniel Salazar 8a6929847d Auth limits, reborn-app tokens, seat addresses, drop-area encode (#4148)
* auth: per-account 2FA code limit, single-use otp-login tokens, recovery mail limits

- /login/otp and /login/recovery-code share a per-account bucket charged on
  wrong codes; an otp-login token completes one sign-in.
- /send-pass-recovery-email adds an IP bucket, a per-account cap and captcha,
  and reuses the outstanding recovery token instead of rotating it.
- Reauth tokens carry the rejection reason; /signup re-attaches a temp
  account only from an expired session.

* auth: app-issued access tokens older than their app stop authenticating

An access token whose row or parent session predates the app now holding its
uid is refused, the same check app and worker sessions already get.

* teams: signup can take an address a seat never confirmed

Password signup, save-account and a provider-verified OIDC sign-up take an
address a team seat holds unconfirmed; the seat's copy is cleared at the
write. A seat address that was confirmed still blocks.

* dev-center: encode dropped directory child names
2026-10-09 00:12:59 -07:00
Daniel Salazar 360f6b1079 fix(events): bill reached deliveries, bound dispatch reads, seal cursors (PUT-1900) (#4146)
* fix(events): bill reached deliveries only, bound dispatch reads, seal cursors

- A broadcast socket copy is billed only when a connection for its holder
  exists on this node, anywhere in this region (connection count) or, for a
  durable row, in a region presence names. Without peers it bills as before.
- Compiled match filters are held in a bounded LRU with a TTL, and dropped
  when another process reports a subscription ended.
- GET /events/fetch reads the continuation from `cursor`, falling back to
  `after`.
- Cursors from /events/fetch, /events/subscriptions and /events/kv-handles
  are encrypted with sealCursor/openCursor; plain cursors still read.
- A durable subscribe counts again after its insert and removes its own row
  when over the cap; the generation bump for that removal is published.
- Subscribe checks access before reading the anchor's owner, and an anchor
  gone in between answers with the subject the caller sent.
- Dispatch reads at most the rows per token routing can use, scanning large
  token hashes; a removal still settles every row on the removed anchor.
- puter.events.unsubscribe() stops routing after the server ends the
  subscription or reports it gone; fetch() accepts `cursor`.

* fix(events): forward kv values only to regions with rows that ask

- A region counts its session rows asking for kv values per token and
  announces the token's value variant (`v#<token>`) over the existing
  `watch` item on the first such row, withdrawing it with the last. Add,
  remove, reap, reanchor and refresh all keep it, reading `includeValue` off
  the row as it is dropped.
- A kv write reads the value variants alongside the watch index and attaches
  the value only to forwards bound for a region that announced one.

* fix(pagination): seal id-keyed cursors on every list endpoint

Apps, subdomains, workers, team (members, directory, teams, audit, own
audit), shares (inbound, outbound), the user-to-user permission audit and
readdir/descendant listings now hand out cursors sealed with sealCursor and
read them with openCursor, so a cursor no longer carries a row id. Plain
cursors issued before still read.

* test(events): pass socket ids to noteConnect in the billing tests
2026-10-09 00:08:13 -07:00
Daniel Salazar 8ecfeec8c3 fix(puter-js): request idle timeout, callback/listener leaks, guarded message handlers (PUT-1908) (#4143)
* fix(puter-js): free one-shot callbacks and listeners, guard message handlers

- setAuthToken starts the GUI cache refresh loop once instead of per call
- IPC reply callbacks are dropped after their reply; dehydrated callbacks stay
- AppConnection removes its window listener when the target app closes
- reply branches for app data and instance count drop their callback
- UI, Debug, tool and GUI-token message listeners no longer reject; an
  unknown or failing tool answers toolResponse with an error
- parseResponse handles a blob response with no content-type
- retry waits remove their abort listener when the wait ends
- EmailConfirmationDialog escapes its message
- EventListener uses a real Map and snapshots listeners per emit
- Context get/set/runWithContext route every known key the same way

* fix(puter-js): time out requests that make no progress

- buildXhr starts an idle clock on send, reset by state changes and body
  progress (upload progress only on requests already carrying a bearer)
- read-safe requests get 60 s while progress is observable, everything
  else 15 min; timeout: 0 on fetchUrl/driverCall/sendWithRetry turns it off
- a timed-out read replays once; writes and AI calls never replay
- timeouts reject with code request_timeout on fetchUrl (TypeError),
  driverCall, streams and the legacy XHR path
- document the limits in rate-limits-and-quotas and the chat stream note
2026-10-09 00:07:20 -07:00
Daniel Salazar ae234ffcc2 fix: only sign-in and picker popups hand the opener a token (#4147)
* fix: only sign-in and picker popups hand the opener a token

Popup actions are an allowlist now: sign-in popups (no action, sign-in,
login, signup) deliver at boot, pickers deliver with the pick, and
request-permission keeps its exchange. Any other action runs no exchange
and posts no token. The consent check in postAuthActions applies to every
action that delivers at boot instead of only no-action and sign-in.

* fix: a dismissed account picker hands the opener no token
2026-10-09 00:05:44 -07:00
Daniel Salazar 2ae2e093e7 keep archive entries inside the extract folder, encode template labels (#4145) 2026-10-08 23:59:33 -07:00
Daniel Salazar dcfb1263de fix(events): single-writer presence items, live-socket counts and region-wide leave grace (PUT-1759) (#4142)
* fix(events): keep each region's presence item written only by that region

- A region that receives a forward for a pair it holds no socket for
  retires its own presence item through the pin-ordered leave path,
  gated by a region-shared claim. The sender no longer retires another
  region's item, and the forward reply no longer carries noSocket.
- Presence reads skip items for regions outside the peer list instead
  of retiring them.
- removeConnection decrements and deletes the count in one script.
- A failed join releases the join pin, and any connect that finds the
  pin free writes the join.
- A single delivery treats a socket on any node of the region as local,
  using the region-wide connection count, and picks the next remote
  candidate by the regions already tried instead of by index.
- Correct the connection-count TTL comment.

* fix(events): count presence by live socket and honor the leave window region-wide

- A region's presence for a pair is a sorted set of socket ids scored by
  last renewal under a new key, replacing the integer count. A socket
  counts while renewed within the concurrency-slot window; add, renew and
  remove each prune lapsed members and report the live count in one
  script, so a node that dies stops holding the pair.
- The last socket going marks the pair as leaving region-wide for the
  grace window, in the same script; a connect clears it. A forward that
  lands on another node inside the window does not retire the item.
- Add the events.presence.write counter, labelled by op (join, leave,
  refresh, retire).
2026-10-08 23:55:55 -07:00
velzie 50e2af34bf feat(puter-js): Add custom streams and perfect negotiation to Peer (#4058)
* add perfect renegotiation

* more advanced peer logic

* add video tracks to peer api

* several fixes

* small fixes, revert e006b1977

* proper reconnect

* fixes

* fixes 2

* fixes 3

* fixes 4

* remove uneeded tests
2026-10-08 23:34:55 -07:00
Daniel Salazar 7adc03cb75 fix: team seats caps (#4141) 2026-10-08 17:50:45 -07:00
dependabot[bot] 807368f7b8 chore(deps): bump handlebars from 4.7.9 to 4.7.10 (#4137)
Bumps [handlebars](https://github.com/handlebars-lang/handlebars.js) from 4.7.9 to 4.7.10.
- [Release notes](https://github.com/handlebars-lang/handlebars.js/releases)
- [Changelog](https://github.com/handlebars-lang/handlebars.js/blob/v4.7.10/release-notes.md)
- [Commits](https://github.com/handlebars-lang/handlebars.js/compare/v4.7.9...v4.7.10)

---
updated-dependencies:
- dependency-name: handlebars
  dependency-version: 4.7.10
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-08 17:20:25 -07:00
Daniel Salazar 4f3a524ace fix: credit usage gui (#4140) 2026-10-08 17:20:00 -07:00
Krish KumarandDaniel Salazar 95b6f86fcc fix(events): resolve relative fs: paths like fs operations do (#4085)
* fix(events): resolve relative fs: paths like fs operations do

* docs(events): use bare relative paths in examples, test both relative forms

* docs(events): clarify that relative fs: subjects come back resolved

---------

Co-authored-by: Daniel Salazar <[email protected]>
2026-10-08 15:23:53 -07:00
Juan Fernando Castro bb829885ef fix: a picker popup hands its opener a token once the user has picked (#4133)
* fix: a picker popup hands its opener a token once the user has picked

A picker opens because the site asked, not because the user agreed to
anything, so the hand-off now waits for the answer the popup was opened
to get. The exchange still runs at boot — the app row and host_app_uid
are what the dialog is opened against — but the token travels with
fileOpenPicked, directoryPicked or fileSaved. Cancelling sends nothing.

deliversTokenAtBoot is the one gate every boot-time hand-off reads, so
the plain exchange, the cross-origin-isolated /login/set path, temp-user
creation and manual signup all follow the same rule. A failed exchange
still reports and closes, having nothing to defer.

* fix: the review's findings on the picker hand-off

Double-clicking a file answers the opener from openItem.js, not from the
UIWindow button, so that pick returned the item and no token. The
deliverer is a global now and every path that answers an opener calls
it: both UIWindow handlers, openItem, and the save flow.

A failed exchange posted `{success: false, token: null}`, and the SDK's
handler sets that as readily as a real token, so a picker that failed
wiped whatever the site already held. The failure now goes only where a
boot-time token would have.
2026-10-08 15:11:47 -04:00
Juan Fernando Castro c55437245b fix: settle the url file-access gate from a grant held on a folder above (#4132)
* fix: settle the url file-access gate from a grant held on a folder above

The gate asks `/auth/check-permissions` with `app_uid`, which resolved
through the permission scan. `fs:` access inherits down a tree and only
ACL walks that, so a grant on a folder never answered for a file inside
it and the gate asked again, writing another per-file row on each Allow.

The `app_uid` branch now routes `fs:<uid>:<mode>` through
`ACLService.check` as the app-under-user actor, which is the same
authority the operation itself will face. Everything else keeps the
scan, including `puter.perms.holds`, which sends no `app_uid`.

* fix: the review's findings on the consent gate

A permission that is not a string reached `startsWith` and faulted the
whole request, where it used to answer false inside the catch; both
parsers now refuse a non-string, which also covers the other callers
that read a request's list.

The path spelling of a permission took the scan while the uid spelling
took ACL, so one file could answer two ways. Both spellings resolve to
their entry and take the same route.

The checks run together rather than one after another: the caller gives
up after five seconds and prompts, and a deep path is several reads.
2026-10-08 15:05:01 -04:00
Juan Fernando Castro d1d55a6bc3 SDK auth plumbing: cache scope, dedupe keys and the token listener (#4112)
* fix: pin the token sender, scope the grant flag, and stop two stragglers

Five of the SDK auth-plumbing items.

**The global `puter.token` handler pinned origin but not source.** Its three
siblings all pin both, and each says why: the GUI origin is shared by every
frame on that domain, so origin alone identifies a domain, not a sender.
`signIn` pins its popup and binds `msg_id`, the consent dialog pins its popup,
and the reauth wait pins the embedder. This one pinned the embedder only when
`env === 'app'`. Since `authenticateWithPuter` now hands off to `signIn`, which
adopts the token itself, the embedder is the only sender this handler is left
serving, so it pins in every environment.

**`is_grant_user_app_permission` was a request-wide flag held across an await.**
It tells the `app-root-dir` rewriter that a row is being written, and while it
is set that rewriter resolves to a real `fs:` permission instead of
`PERMISSION_FOR_NOTHING_IN_PARTICULAR`. Anything evaluating an `app-root-dir:`
permission beside the grant -- the grant route runs one under `Promise.all`
with fs work -- read the same flag. It now lives in a scope derived for that
call, so what the write needs to say about itself is not also said to its
neighbours. `runInDerivedContext` is new: `runWithContext` starts an empty
store, which would drop the `actor` the rewriter reads.

**`fs.stat` and `fs.readdir` deduped on parameters alone**, so two identities
in one realm could share a result. They now key on origin and token as
`os/user.js` does, which documents the reason.

**`signIn`'s relay poll never stopped.** Closing the popup settles the promise;
the loop kept asking `/login/wait` every second for the life of the page.

**`triggerReauth`'s five-minute timeout was never cleared**, so a successful
reauth left a timer to fire into a settled promise.

The concurrency test parks the rewrite and reads the flag while it is held --
my first attempt read before the flag was even set and passed either way.

* fix: scope the fs cache per identity, and retarget two paths that never ran

The remaining SDK auth-plumbing items.

**`puter._cache` outlived the identity that filled it.** `_clearAuthToken`
dropped the token and the stored session but left the cache and `whoami` as
they were, and its keys were raw paths -- `item:/alice/Documents` and a
`~`-relative key alike, identical whoever asked. A GUI user switch without a
reload could be served the previous user's stat or readdir. Keys now carry the
origin and token they were read as, built in one place so the writers, the
readers, the invalidations and the warm-up probes cannot drift apart, and
signing out drops the cache and `whoami` with the token.

**The driver permission prompt could not fire.** Both paths waited for a 200
carrying `{success:false, error:{code:'permission_denied'}}`. The driver route
answers `success: true` or throws, so a refusal arrives as a 403 -- the shape
the retry now reads. Its two tests synthesised the old envelope, so they passed
against a response the API does not send; they use the real one now, and the
prompt goes from never called to called once.

This is a behaviour change: a driver call refused for permissions will now
prompt, which is what the path was written to do. Happy to delete it instead if
the prompt is not wanted.

The half-written copy of the same idea in `utils.js` is gone -- it prompted for
one hard-coded interface and its retry was a `// todo`.

**The verification-gate single-flight was keyed on nothing**, so a phone-gated
request in flight beside an email-gated one took the email answer, replayed,
failed again and surfaced raw without ever asking. Keyed per gate: same gate
still shares one dialog, a different gate gets its own.

* test: ask the cache the same way the code does

The API suites built `item:<path>` keys by hand, so they stopped finding
anything once the real keys carried the identity they were read as. They go
through `fsCacheKey` now, which is the point: a test that spells the key itself
passes until the moment the spelling changes, and then fails for a reason that
has nothing to do with the behaviour it is guarding.

These run against a live server, so a plain `vitest run` does not reach them --
they were green locally and red in CI.

* fix: the review's findings on this branch

Eight, and three of them were regressions this branch introduced.

**Pinning the token sender to `globalThis.parent` in every environment broke
the case it was meant to protect.** On a top-level page `parent` is the page
itself, and the token arrives from a popup, so the handler stopped accepting
anything. The GUI posts `puter.token` to `window.opener` for the file pickers
too, not only for sign-in, and this handler is their only consumer -- a site
calling `showOpenFilePicker()` would have signed the user in, dropped the token
and run the following read with none. It now pins the embedder when framed and
a window this SDK opened when not; `UI.js` already tracked its picker popups
for the same reason, so that set is shared rather than duplicated.

**The GUI still spelled a cache key by hand.** `updateSubdomainsForItems` built
`item:<path>`, which no longer matches what the SDK writes, so its `get` could
not hit and its `set` wrote an orphan nothing reads and nothing evicts -- a
published website's badge quietly stopped appearing on a cached render. It goes
through `fsCacheKey` now, and that is the last hand-spelled key in the tree.

**Retargeting the driver permission prompt to 403 was wrong and is reverted to
the deletion the ticket asked for.** The route refuses with `forbidden`, not
`permission_denied`, so the retarget still missed it; meanwhile a driver that
raises `permission_denied` for its own reasons would have opened a prompt that
had nothing to do with the refusal. And the prompt asks for
`driver:<iface>:<method>` while the route checks `service:<driver>:ii:<iface>`,
so granting it changes nothing. The path cannot work as written.

The rest were pre-existing or latent:

`cacheWhoami_` wrote its answer without checking the token it was issued under
was still current, so a reauth landing mid-flight could repopulate `whoami`
with the previous identity. `_dropIdentityCaches` also left `whoamiCache_`,
which other call sites await.

`setAuthToken` now drops the identity caches when the token actually changes:
keys carry the token and the store is persisted with no TTL, so a rotation that
did not go through `_clearAuthToken` orphaned a whole key space with nothing to
reclaim it.

The verification-gate entry was registered after the body could already have
deleted it -- a synchronous throw stranded a resolved "not verified" in the map
for the life of the page. It registers first and clears on a microtask.

`signIn`'s poll is bounded as well as settle-checked: on a cross-origin
isolated page opened from a gesture, nothing watches the popup, so closing it
never settled the promise and the loop ran on exactly as before.

`runInDerivedContext` says in its docblock that callee writes do not survive;
only the one flag needs the scope, and a future caller should not expect to
hand a value back through it.

* fix: the review's findings on the SDK auth plumbing

**The relay bound stopped asking without saying so.** The deadline left the
`signIn` promise pending and its listener attached. On an isolated page opened
from a gesture the relay is the only thing that can settle it -- nothing watches
the popup there -- so a user who took longer than the deadline was left with a
promise that never resolved, and `authenticateWithPuter` queues every later
implicit-auth call behind it. It rejects now.

**Pinning the token source left `signIn`'s own popup untracked.** Only the
picker popups registered, so on a top-level page the global handler bailed on a
sign-in it had opened itself: `puter.onAuth` never fired, and
`puterAuthState.authGranted` was never set. `openAuthPopup` is the one place
auth popups are opened, so they register there.

**A closed popup is ours too.** Rejecting one that had already closed loses a
message from a popup that posts and then closes itself, which is why `UI.js`
tolerates the same case. Kept, with the set bounded instead.

**The gate key carried the factors**, so one route naming them and another not
opened two dialogs for the same gate. It keys on the gate, as `ctx.done` does.

**`fsCacheKey` left `~` unresolved**, so a writer and an invalidation spelling
the same place differently missed each other -- the drift the one key builder
was meant to end.

The dead `opts.permission` went with the path that used to read it.

Four workerd-runner failures are left: `ai > chat stream adapter supports
cancellation`, `kv > a GUI boot key written by another client reads fresh` and
two cache-warmer ones. All four fail the same way on `origin/main`, and the node
and browser runners pass.

* chore: one-line the comments added here

* chore: reflow the comments the removed retry cause left ragged
2026-10-08 15:04:49 -04:00
Michael Lip 4ea93e6a73 fix(i18n): add GUI keys that rendered as raw key names (#4074)
The 2FA spinner showed "verifying", the files selection bar showed a
"done" button, and two login and signup errors showed their key names,
because i18n() returns the key itself when en.js has no entry for it.
The login error now uses the existing login_email_username_required key,
which already has translations. A test checks that every literal i18n()
key in the GUI exists in en.js.
2026-10-08 11:42:07 -07:00
Daniel Salazar 80736de0f1 test(events): wait for the recorded failure, not any future lease score (#4127)
The in-flight claim's lease (2x the invoke timeout) is also a future score, so
polling heldForMs > 0 could pass before the failed attempt's hold was written:
the backoff test read 600 instead of 2000, and the 404/hang/429/suspended-owner
tests ended while their attempt was still in flight, letting it land in the
next test. Wait on the subscription's failure counter instead, which only moves
after the hold or gap marker is written.
2026-10-08 11:39:06 -07:00
dependabot[bot] 085ab1774f chore(deps): bump shell-quote from 1.10.0 to 1.12.0 in /src/puter-js (#4120)
Bumps [shell-quote](https://github.com/ljharb/shell-quote) from 1.10.0 to 1.12.0.
- [Changelog](https://github.com/ljharb/shell-quote/blob/main/CHANGELOG.md)
- [Commits](https://github.com/ljharb/shell-quote/compare/v1.10.0...v1.12.0)

---
updated-dependencies:
- dependency-name: shell-quote
  dependency-version: 1.12.0
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-08 11:32:57 -07:00
dependabot[bot] 505fd13d97 chore(deps): bump shell-quote from 1.9.0 to 1.12.0 (#4121)
Bumps [shell-quote](https://github.com/ljharb/shell-quote) from 1.9.0 to 1.12.0.
- [Changelog](https://github.com/ljharb/shell-quote/blob/main/CHANGELOG.md)
- [Commits](https://github.com/ljharb/shell-quote/compare/v1.9.0...v1.12.0)

---
updated-dependencies:
- dependency-name: shell-quote
  dependency-version: 1.12.0
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-08 11:30:52 -07:00
Reynaldi Chernando 47a1be7f70 Rename "Add Existing Account" menu item to "Add Another Account" (#4124) 2026-10-08 11:25:52 -07:00
Filip Kujundžić 38c7191009 Merge pull request #4135 from HeyPuter/feat/claude-haiku-5-5 2026-10-08 19:56:01 +02:00
404oops b6f31e3b02 feat(ai): add Claude Haiku 5.5 2026-10-08 19:44:17 +02:00
Filip Kujundžić 3c852cf9cf Merge pull request #4134 from HeyPuter/feat/mistral-large-4-sonnet-pricing
feat(ai): add Mistral Large 4 and update model pricing
2026-10-08 19:37:53 +02:00
404oops 0afe8ccfdd docs(ai): keep model update documentation focused 2026-10-08 19:29:36 +02:00
404oops 266390d23d feat(ai): add Mistral Large 4 and update model pricing 2026-10-08 19:26:06 +02:00
Filip Kujundžić b6150d63f3 Merge pull request #4090 from HeyPuter/fix/ai-stream-cancellation
fix(ai): propagate cancellation and stream termination failures
2026-10-08 13:33:50 +02:00
404oops 4e8277ea4b docs(ai): clarify streaming errors and stopping streams 2026-10-08 13:21:38 +02:00
Daniel Salazar b6dfc6f903 fix: harden card verification (#4126) 2026-10-07 23:11:49 -07:00
dependabot[bot] 3d7deaa403 chore(deps): bump smtp-server from 3.19.13 to 3.19.17 (#4083)
Bumps [smtp-server](https://github.com/nodemailer/smtp-server) from 3.19.13 to 3.19.17.
- [Release notes](https://github.com/nodemailer/smtp-server/releases)
- [Changelog](https://github.com/nodemailer/smtp-server/blob/master/CHANGELOG.md)
- [Commits](https://github.com/nodemailer/smtp-server/compare/v3.19.13...v3.19.17)

---
updated-dependencies:
- dependency-name: smtp-server
  dependency-version: 3.19.16
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-07 22:36:57 -07:00
Daniel Salazar 199d72cbdf fix(events): refuse scoped access tokens on durable events routes (PUT-2048) (#4105)
Durable subscribe, list, fetch and unsubscribe now apply the same scoped
token check as the handler, worker and kv-handle routes:
`isAccessTokenActor(actor) && !isAccountContext(actor)`. An access token
an app issued (effectiveApp = the app) is treated the same as one the
account issued: subscribe answers 403 `events_durable_requires_account`,
list and fetch return an empty page, unsubscribe answers 404.

`rowInActorScope` returns false for these tokens, which covers durable
unsubscribe and ack. The session verbs that also use it already refuse
access tokens at the socket handshake.

Full-access tokens, app tokens and sessions are unchanged.
2026-10-07 22:35:09 -07:00
Daniel Salazar f234bd477c fix(auth): answer a username lost to a concurrent write with the taken-name 400 (#4115)
The duplicate-username check in the signup placeholder claim, change-username
and save-account runs before the user-row write, so a concurrent write can
take the name in between and the unique index rejects ours. Each path now
re-reads the holder from the primary on a unique violation and returns the
400 its pre-check returns for a taken name, instead of a 500. In all three,
the home-directory rename, rename budget charge and username-changed event
run after the user-row write, so nothing has happened yet when it fails.
2026-10-07 22:34:31 -07:00
Daniel Salazar 2638e9ce93 fix(kv): batch-read GUI boot keys only when the SDK runs as the GUI (PUT-1899) (#4117)
The boot-key batch now starts only when puter.env is 'gui'. In every other
environment a get() of one of those names reads that key on its own, like
any other key, instead of fetching all 14 and serving them for 4 s.
2026-10-07 22:34:22 -07:00
dependabot[bot] accf33a50e chore(deps-dev): bump source-map-js from 1.2.1 to 1.2.2 (#4123)
Bumps [source-map-js](https://github.com/7rulnik/source-map-js) from 1.2.1 to 1.2.2.
- [Release notes](https://github.com/7rulnik/source-map-js/releases)
- [Changelog](https://github.com/7rulnik/source-map-js/blob/main/CHANGELOG.md)
- [Commits](https://github.com/7rulnik/source-map-js/compare/v1.2.1...v1.2.2)

---
updated-dependencies:
- dependency-name: source-map-js
  dependency-version: 1.2.2
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-07 17:14:30 -07:00
dependabot[bot] ab175b0fb4 chore(deps): bump proxy-addr from 2.0.7 to 2.0.8 (#4122)
Bumps [proxy-addr](https://github.com/jshttp/proxy-addr) from 2.0.7 to 2.0.8.
- [Release notes](https://github.com/jshttp/proxy-addr/releases)
- [Changelog](https://github.com/jshttp/proxy-addr/blob/master/HISTORY.md)
- [Commits](https://github.com/jshttp/proxy-addr/compare/v2.0.7...v2.0.8)

---
updated-dependencies:
- dependency-name: proxy-addr
  dependency-version: 2.0.8
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-07 17:13:23 -07:00
dependabot[bot] 3fcbd895a7 chore(deps): bump sharp from 0.35.4 to 0.35.5 (#4081)
Bumps [sharp](https://github.com/lovell/sharp) from 0.35.4 to 0.35.5.
- [Release notes](https://github.com/lovell/sharp/releases)
- [Commits](https://github.com/lovell/sharp/compare/v0.35.4...v0.35.5)

---
updated-dependencies:
- dependency-name: sharp
  dependency-version: 0.35.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-07 16:12:01 -07:00
dependabot[bot] 4c9e4d8e79 chore(deps): bump mysql2 from 3.24.4 to 3.24.5 (#4082)
Bumps [mysql2](https://github.com/sidorares/node-mysql2) from 3.24.4 to 3.24.5.
- [Release notes](https://github.com/sidorares/node-mysql2/releases)
- [Changelog](https://github.com/sidorares/node-mysql2/blob/master/Changelog.md)
- [Commits](https://github.com/sidorares/node-mysql2/compare/v3.24.4...v3.24.5)

---
updated-dependencies:
- dependency-name: mysql2
  dependency-version: 3.24.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-07 16:11:46 -07:00
Juan Fernando Castro cf574e4b34 Merge pull request #4110 from HeyPuter/juancastro/icloud-dot-significance
Treat dots as significant on Apple's mail domains
2026-10-07 17:11:59 -04:00
Daniel Salazar 2e275a8f6b fix: godmode installation (#4116)
* fix: godmode installation

* fix: godmode classifier

* fix: vitest
2026-10-07 13:16:10 -07:00
Nariman Jelveh b5bd917119 Merge branch 'main' of https://github.com/HeyPuter/puter 2026-10-07 12:39:47 -07:00
Nariman Jelveh 3999dd062f Update RecommendedAppsService.ts 2026-10-07 12:39:34 -07:00
Daniel Salazar 3e234d7329 fix: 402 error messaging (#4114) 2026-10-07 12:34:45 -07:00
Daniel Salazar f674e51b4d fix: answer lost write races with 4xx, and time health-check timeouts against loop stalls (#4107)
- signup: a username taken by a concurrent signup between the duplicate
  check and the insert now returns the pre-check's 400 instead of
  surfacing the unique-key error as a 500. The holder is read from the
  primary.
- fs move: when the UPDATE trips the (parent_id, name) unique key after
  the collision probe missed the occupant, re-read it from the primary and
  resolve it the same way the probe would have (overwrite, dedupe, or the
  409 item_with_same_name_exists), then retry the move once.
- server health: a check that hits the per-check timeout reports how long
  the event loop was blocked during it, like the database-liveness latency
  failure already does.
2026-10-07 12:02:05 -07:00
Daniel Salazar d395a3aa94 fix(kv): page through flush, cap per-call batch sizes, validate decr and del (#4106)
- flush follows LastEvaluatedKey until the namespace is empty and rejects
  when a batch delete fails; cached reads for every key a page returned are
  still invalidated and the flushed event still fires.
- decr validates the amount map before negating it, so it accepts exactly
  what incr accepts.
- del checks the key like every other store method (oversized key is a 400).
- list offset emulation treats a missing continuation key as end of data
  instead of restarting from the first key.
- Array get maps results through a Map instead of a linear find per key.
- The driver gates includeTotal on credit like value listings.
- New driver caps in drivers/kv/limits.ts: 1,000 keys per array get, 1,000
  items per batchPut (both rejected past the cap), list limit lowered to
  1,000. Published in rate-limits-and-quotas.md and the set/list pages.
2026-10-07 12:00:50 -07:00
Daniel Salazar fde323842e fix(kv): SDK list rejections, boot-cache read-after-write, byte-accurate size checks (PUT-1899) (#4108)
- puter.kv.list() no longer throws synchronously for invalid options; it
  returns a rejected promise, or for stream: true an iterator whose first
  next() rejects. Error codes unchanged.
- Writes through puter.kv (set, del, incr, decr, add, remove, update,
  expire, expireAt, flush) stop the boot-key batch from serving the keys
  they touch, so the next get() reads the store. Untouched boot keys are
  still served from the batch.
- Client-side key/value size checks measure UTF-8 bytes, and values are
  measured as their JSON encoding, matching the store. Same codes and
  messages.
2026-10-07 11:48:45 -07:00
Daniel Salazar 4a54beac6b fix(fs): SDK uid stat cache, readdir page types, move lookup errors, UID-shaped paths (#4109)
- stat: only path-addressed results are cached; uid-addressed stats no
  longer share one cache key.
- readdir: overloads declare the page result for recursive + limit/offset
  and for the positional page/stream forms.
- move: only a not-found destination lookup means "new path"; other
  lookup failures reject. A uid destination skips the lookup.
- getAbsolutePathForApp always resolves paths. read/copy/move/delete use
  getAbsolutePathOrUidForApp, which keeps sending UID-shaped strings as uids.
- Docs for read, copy, move, delete and readdir.
2026-10-07 11:46:22 -07:00
Juan Castro 0555ea6479 Merge branch 'main' into juancastro/icloud-dot-significance 2026-10-07 11:01:12 -04:00
Juan Fernando Castro 27eebc7300 Merge pull request #4086 from HeyPuter/juancastro/permissions-access-control-cleanup
Permissions and access-control cleanup
2026-10-07 10:57:41 -04:00
Juan Castro 4b5f509066 fix: only rewrite rows the old rule actually wrote, and fix the invites too
Two findings from review, both of which contradict what the last commit
claimed.

**The backfill could merge keys, not only split them.** I asserted it "can only
split rows apart, so it cannot collide". Not true, and reproducible: a row
predating the `clean_email` column has a NULL there, so its index key is
`lower(email)` -- filling it in folds `+tag`, and `[email protected]` lands on a
key `[email protected]` already owns. A legacy row holding an un-lowercased
`clean_email` collides the same way once lowercased. Either throws, and the two
engines that re-run every file on every boot would then refuse to start until
someone operated on the database by hand.

Both updates now fire only where the stored value is exactly what the old rule
produced and differs from the correct one. That makes "splits only" true: the
sole change is dots coming back. NULLs and legacy values are left alone.

The same predicate makes them write-idempotent. Postgres has no no-op-update
suppression, so without it every boot rewrote a tuple per Apple row forever.

**The pending-invite residue misdelivers rather than just going missing.**
`claimPendingShares` looks the row up by the canonical address, so whoever
confirms `[email protected]` claims the invites addressed to
`[email protected]` -- someone else's file, handed over on confirmation. I had
written this off in the PR as unfindable invites that "expire on their own";
`share` has no expiry column and the lookup has no time predicate, so they are
claimable indefinitely.

Fixed here rather than later: the typed address is kept as `invitedAddress`
whenever it differed from the canonical, which is exactly the collapsed rows,
so the invites recompute from it under the same guard.

Verified by seeding a database at the old version with the two rows that used
to abort the migration, and booting: it reaches version 87, repairs the
collapsed row, and leaves the NULL and legacy rows untouched. Replaying the
statements changes zero rows.
2026-10-07 10:23:30 -04:00
Juan Castro ece8ad5f1d fix: register the Apple-domain backfill with the sqlite runner
The sqlite migration list is hand-maintained, so the file alone did nothing —
a fresh boot still reported 87 files and stopped at version 86. MySQL and
Postgres read their directories, so only this one needed the entry.

Caught by booting a local instance rather than by the suite: the client tests
pin `CURRENT_SCHEMA_VERSION`, which fails once the entry exists but says
nothing while the file is simply unregistered.

Verified against a seeded database at the old version. `[email protected]`
went from `[email protected]` to `[email protected]`, `[email protected]`
from `[email protected]` to `[email protected]` — subaddressing still folded — and a gmail row
was left alone.
2026-10-07 09:45:26 -04:00
Juan Castro cf817b0f0a fix: treat dots as significant on Apple's mail domains
`cleanEmail()` applied gmail's `dots_dont_matter` rule to icloud.com, me.com
and mac.com. Apple allocates the exact string: `[email protected]` and
`[email protected]` are two mailboxes with two owners. Collapsing them made one
canonical address stand for both.

What that canonical address feeds:

- `findEmailOwner` falls back to it, so an OIDC sign-in whose verified claim is
  the dotted address resolves to the account holding the undotted one, and
  `linkProviderToUser` has nothing left to object to -- the provider verified a
  real mailbox and the account's address is confirmed.
- The unique index on COALESCE(clean_email, lower(email)) reads it, so the
  rightful owner of a dotted Apple address is refused signup because the
  undotted account already "owns" it.
- Pending share invites are stored and claimed on it.

Subaddressing is untouched: Apple does fold `+tag`, so that rule stays. Gmail
keeps both rules -- there the fold is correct.

A migration recomputes `clean_email` for those domains on all three engines,
since rows written before this still hold the collapsed value and the index
reads the stored one. Verified the SQL against the function rather than by
inspection: same five inputs through `cleanEmail()` and through sqlite give
byte-identical output. It can only split rows apart, never merge them, so it
cannot collide with the unique index.

Not changed: `findUserByEmail` still matches on the canonical address. The
cross-match it documents is correct where the provider really does fold --
`[email protected]` is the same inbox as `[email protected]`. Requiring an
exact match would break that for no benefit here; the fix is for the data to
say what the provider does.
2026-10-07 09:17:03 -04:00
Juan Castro 6efb96d560 Merge branch 'juancastro/flat-permission-view-scope' into juancastro/permissions-access-control-cleanup 2026-10-07 09:00:40 -04:00
Juan Castro 5596e70f25 Merge branch 'main' into juancastro/flat-permission-view-scope 2026-10-07 09:00:18 -04:00
Daniel Salazar e477bca3eb test(events): wait for the failure hold, not any future lease score
The in-flight claim's lease (2x the invoke timeout) is also a future score,
so polling for "held" could read it before the failed attempt's backoff
hold was written. Wait for deferAfterFailure to settle instead.
2026-10-06 22:32:02 -07:00
Daniel Salazar 11e48c68a9 fix: verification for sms (#4101) 2026-10-06 22:03:07 -07:00
Daniel Salazar 464f758a97 fix: verification for sms (#4100) 2026-10-06 22:00:14 -07:00
Daniel Salazar fadee6fac5 fix: verification for sms (#4099) 2026-10-06 21:59:58 -07:00
Daniel Salazar 6bce256589 refactor(ai): trim redundant cancellation checks
- Chat driver runs each attempt through one helper and checks the abort
  signal only on entry and in the attempt catch blocks, where it keeps a
  cancelled request from marking the route unhealthy.
- A stream ended after cancellation still emits ai.prompt.cost-calculated;
  only ai.prompt.complete is skipped.
- pipeNdjsonStream owns its line buffer and onEnd is optional; drop the
  errored guards that can no longer fire.
- Drop the Claude post-loop abort check (finalMessage already throws).
- Skip provider error logs when the caller cancelled.
- Simplify the usage_limited cap condition.
2026-10-06 19:49:25 -07:00
Daniel Salazar d2cdde5147 fix: verification for sms (#4098) 2026-10-06 19:36:18 -07:00
Daniel Salazar 50ee0818ea feat: trial cleanup (#4097) 2026-10-06 18:20:18 -07:00
Juan Castro ad1be70d06 Merge branch 'juancastro/flat-permission-view-scope' into juancastro/permissions-access-control-cleanup 2026-10-06 19:44:37 -04:00
Juan Castro 6277f106c4 Merge branch 'main' into juancastro/flat-permission-view-scope 2026-10-06 19:44:19 -04:00
Juan Castro 738a8058ef fix: close the path oracle where it actually was
Local testing against a running instance found the service-side fix was not
enough. Both answers were 403, but they did not read the same:

  exists  -> permission_denied: fs:/other/rd:read
  missing -> Cannot create outside your home directory   (forbidden)

The second comes from `#planFsPathCreate` in the controller, which probes the
path itself and only calls the create guard once it knows the path is absent.
Its comment said so outright -- checked "only once the path is confirmed
missing, so an existing shared path outside the caller's home is not a new
rejection" -- which is the oracle stated as an intention.

A foreign path now refuses word for word whichever it is. Responses are
byte-identical apart from the caller's own input.

`rewritePermissionForActor` collapses the create guard's `forbidden` the same
way it already collapsed the 404, for the paths that reach it through the
service rather than the controller.

The existing test pinned `forbidden` here; its subject is that nothing is
created in someone else's tree, which still holds. Alongside it is the
invariant itself: refuse a present and an absent foreign path, mask the path,
compare the whole error. Without the controller check that fails.
2026-10-06 19:41:55 -04:00
Juan Fernando Castro 0e2f9663b2 Merge pull request #4091 from HeyPuter/juancastro/teams-cleanup
Teams cleanup: pagination, resumable delete, and credentials that stop leaving by mail
2026-10-06 19:27:09 -04:00
Juan Castro b8fcbffe3f fix: the five remaining review findings
**The path oracle was not closed.** The rewrite only remapped the *error*, so
success-vs-403 still answered whether a foreign path exists. Narrower than
reported: every other grant path already gates on `canManagePermission`, and
only `grantUserAppPermission` has no authority check -- deliberately, since
granting your own app something you don't hold is inert. It now refuses a
foreign path the actor cannot see, with the same error a missing one gives.

My first test for this passed without the fix: `makeUserActor` creates no
fsentries, so both paths 404'd and nothing distinguished them. It inserts a
real entry now, and the control fails with "expected null to match 403" --
null being the grant that succeeded, which is the oracle.

**`#validateEmail` still failed open** on the main signup / change-email /
confirm path while its OIDC twin was hardened. Same shape as OIDC now: a gate
that did not run is not a pass.

**The 16-permission cap broke `puter.perms.check()`** for longer lists. The
SDK's own docblock promised "one round trip however many are asked about",
which the cap made untrue, so it batches instead. The reported second half --
the UI grant poller burning its timeout -- is not reachable:
`MAX_REQUESTED_PERMISSIONS` already caps that path at the same 16.

**`listener_failed` was set by wildcard listeners.** An unrelated observer on
`email.*` throwing would refuse every signup. Only an exact-key listener
implements the event, so only that one speaks for the gate.

**`whoami()` labelled everything `Unauthorized`.** A 502 with an HTML body is
not a 401; the server's `code` now rides along and the fallback names the
status.
2026-10-06 19:23:40 -04:00
Juan Castro 755b58fd49 fix: don't hand the owner's subtree size to a see-only caller
Switching `getSubtreeSize` to `entry.userId` fixed a recipient reading 0 for a
shared folder, but `/stat` is gated on `see` and `return_size` is caller-
supplied. `see` is the weakest level -- the folder is visible, what is under it
is not -- so the sum now covered entries the caller had no way to enumerate.

It is answered only for a caller who passes a `list` check, which is where the
contents become enumerable anyway. Everyone below that gets the directory row's
own size, which is what they got before this branch.

`#assertAccess` grew a non-throwing sibling for this; the descriptor both need
moves into `#aclDescriptor`.

Test grants `see`, asserts 0, grants `list`, asserts the real size. Without the
gate the `see` case reads the file's bytes.
2026-10-06 19:08:46 -04:00
Juan Castro bd5aa901ab perf: batch the per-page reads in a team delete
Measured first, against a real server, counting DB round trips: a 40-seat
delete was 284 queries, 7.1 per seat, scaling linearly. Four of those seven
answered for the whole page and had no reason to be asked per account.

  120x SELECT * FROM `user` WHERE `id` = ?
   41x INSERT INTO `audit_team_membership`
   40x SELECT COALESCE(SUM(size), 0) FROM fsentries
   40x UPDATE `user` SET `suspended` = ?
   40x SELECT `uuid` FROM `sessions` WHERE `user_id` = ?

The page now reads who is already suspended, what each holds, and the rows the
notices need in one query each, and writes the audit as one statement. What is
genuinely per account -- the suspension, its billing event, the mail -- stays
per account under the same concurrency bound.

Same 40-seat delete is now 170 queries, 4.3 per seat.

Three new store methods rather than raw SQL at the call site:

`getSuspendedByIds` reads the primary, deliberately not `getByIds`, which is
cache-first. A stale `suspended` is exactly what re-charges a seat on a retry,
which is the thing this path was just fixed to stop doing.

`getHeldBytesForUsers` is `getHeldBytes` with a GROUP BY.

`appendAuditMany` is `appendAudit` as one multi-row insert, still written
before any suspension, so a failed append cannot leave an unlogged one.

What is left is concentrated in `#suspend`: the UPDATE, the forced re-read the
user store does afterwards to refresh its cache, and the session drop. Those
are shared store paths and batching them means owning their cache invalidation,
so they are not in scope here.
2026-10-06 18:55:53 -04:00
Juan Castro b84fc6c158 Merge branch 'juancastro/flat-permission-view-scope' into juancastro/permissions-access-control-cleanup 2026-10-06 18:26:31 -04:00
Juan Castro edb0c2a712 Merge branch 'main' into juancastro/flat-permission-view-scope 2026-10-06 18:13:27 -04:00
Juan Castro 55e29666d0 Merge branch 'juancastro/flat-permission-view-scope' into juancastro/permissions-access-control-cleanup 2026-10-06 18:13:02 -04:00
Juan Castro 66774ff90c Merge branch 'main' into juancastro/permissions-access-control-cleanup 2026-10-06 18:12:39 -04:00
Juan Castro 0fbb6def5a Merge branch 'main' into juancastro/flat-permission-view-scope 2026-10-06 18:12:25 -04:00
Juan Fernando Castro d9ee207ed0 Merge pull request #4092 from HeyPuter/juancastro/storable-email-gate
Gate an email address once, before it is stored
2026-10-06 18:10:19 -04:00
Juan Castro 0495ea0f9d fix: close the holes the review found in the storage gate
Four, all real.

The gate threw on a non-string. `validator.isEmail` asserts its argument, and
two call sites hand it an unvalidated body value -- `send-pass-recovery-email`
is public, so `{"email": 1}` was a 500 rather than a 400. The check belongs in
the gate, not in each caller, which is the whole premise of having one.

OIDC signup never shape-checked the claim at all. `claims.email` went from a
truthiness check straight into `user.create`, so a provider we do not run could
store exactly the string this branch exists to keep out. `email_verified`
attests the mailbox, not the characters.

The change-email confirm handler promoted the staged column without re-reading
it through anything. A row staged before this deploys still lands on `email`;
one check at promotion closes that window.

The invite path kept its own regex, and it was never the same rule. It admitted
`[email protected]`, `.[email protected]`, `[email protected]`, `a;[email protected]` and `a,[email protected]` -- the last of
which nodemailer reads as two recipients -- each one a permanent pending row
that spent daily quota, which is the failure the comment above the constant
said it existed to prevent. It uses the gate now, and the second regex is gone.

A regression test each, all four confirmed to fail without their fix.
2026-10-06 17:20:35 -04:00
Juan Castro 73698be964 fix: one gate for an address we are about to store
`validator.isEmail` accepts a quoted local part, so an address whose local part
is an arbitrary string passes it. Five call sites relied on that check alone
before writing the address to `user.email`, and the invite path's own shape
regex was equally permissive.

`isStorableEmail` is now the single gate: the library check, plus a refusal of
`<`, `>` and `"`, which are legal in an address only inside the quoted form.
Nothing deliverable uses that form, so the rule costs no real address --
apostrophes and ampersands in a local part still pass, and the suite covers
them.

One function rather than a check each caller has to remember, because the next
place that stores an address will not remember.
2026-10-06 16:47:15 -04:00
Juan Castro 25f132b28d fix: address code review on the teams cleanup
Six findings, all real.

`total` described rows the page withheld. The count ran over every live team the
user belongs to, while the app-actor filter ran afterwards on the items only, so
`GET /teams?includeTotal=true` from an app returned `items: []` with a non-zero
`total` -- the exact thing the filter exists to prevent, and newly reachable
because `includeTotal` was inert before this branch. The directory-open
restriction moves into the query, so items, cursor and total all describe the
same set.

The GUI told owners the opposite of what the server now does: "we email the
username and temporary password to it." An owner who believed it would dismiss
the one copy of the password and strand the seat. Rewritten.

`runWithConcurrencyLimit` rejects on the first error but cannot cancel the
workers beside it, so a failed seat let up to seven others finish suspending and
emailing "your team has been closed" for a team that was still live. The page
now settles before the rejection is rethrown, which leaves a state the retry can
resume from instead of an arbitrary one. My earlier claim that failure
semantics were unchanged was wrong.

`#disableSeatForDelete` returned `true` on both paths, so the boolean read as if
it distinguished them. It returns nothing; the counter uses the settled results.

`listTeamsForUser` had picked up a second stacked docblock -- the same defect
this branch removes from `#cached` one file over.

`reissueCredential` sent the provisioning mail, which after the credential
change was a byte-identical duplicate saying nothing about the password it had
just invalidated, while the GUI promises the member hears about it. It has its
own template now.

Regression tests for the count leak and the reissue notice; both confirmed to
fail without their fix.
2026-10-06 16:22:48 -04:00
Juan Castro 36a22b05a2 docs: stop claiming the directory discloses what /members does
Both docblocks on the directory route said it carries nothing a colleague
cannot already read through `/members`. It carries `uuid`, and `/members`
withholds `uuid` from everyone but the owner, with its own comment pointing
apps at the directory to get one.

The disclosure itself is the opt-in working as designed: `directory_enabled` is
the owner's consent to expose a roster, every member reaches the route the same
way whether they call it from a session or through an app, and the SDK and the
teams recipes already document the directory as where a stable id comes from.
So the comments are what was wrong, not the route.

Removing `uuid` from the directory would be the other way to make them agree,
but it breaks `listDirectory()` for every caller and contradicts the published
recipes; that wants a decision, not a cleanup commit.
2026-10-06 16:02:13 -04:00
Juan Castro 9d8e37dc6d fix: stop an unconfirmed claim on an address from blocking invites to it
`#resolveRecipient` already refuses to treat an unconfirmed address as an
identity, and says why: "resolving it would hand the share to whoever registered
the address first." `#invite` then looked the same address up again and honoured
whatever that row said about blocking -- `blocksAllShares`, or a block on this
particular sender.

So registering an address you do not own, and never confirming it, was enough to
refuse every invite anyone sends to that mailbox. Nothing about that row is
evidence the mailbox is yours, which is the premise resolution already works
from. A team seat reaches this cheaply: those accounts are provisioned with
`requires_email_confirmation: false` and an address the owner typed.

The block is now honoured only for a row that confirmed the address. The alias
reasoning behind the lookup is unchanged -- a confirmed holder still cannot be
reached through a variant they did not type.

Not addressed here: the same unconfirmed seat still holds the address against
the real owner's signup, because `idx_user_owned_email` keys on having a
password rather than on confirmation. Narrowing that is a schema change and
wants its own ticket.
2026-10-06 16:00:32 -04:00
Juan Castro 6826071156 fix: stop mailing a plaintext team credential to an unconfirmed address
Provisioning a seat lets the owner supply any `email` that passes a format
check. Puter then mailed a working temporary password to it. Nothing confirms
the owner controls that address -- provisioned accounts are created with
`requires_email_confirmation: false`, because the team is the trust anchor for
the account, not for the mailbox. `resendActivation` reissues the credential and
mails it again, up to the 20/day `team:reset-password` budget, so the same
address can be hit repeatedly.

The credential now stays out of the mail on both paths, which is what
`resetMemberPassword` in the same file already does. Nothing is lost: all three
routes return `temporary_password` to the owner, and the SDK has always
documented that as the delivery -- "The returned password is shown once and is
not retrievable afterwards -- deliver it out of band." The mail still goes out,
naming the account and saying the team will supply the password.

The two tests that pinned the old shape are inverted rather than deleted; one of
them carried the rationale "without it this is the only copy", which the return
value contradicts.
2026-10-06 15:56:33 -04:00
Juan Castro d896250ae2 fix: make team deletion resumable, and stop it serializing its mail
`deleteTeam` tore seats down one at a time, each one awaiting a mail send, so a
full 40-seat team was 40 round trips end to end inside a single request. The
per-seat work is independent, so it now runs under the same bounded-concurrency
helper the rest of the backend uses for fan-out sends.

The loop also had no equivalent of the guard `disableMember` carries. That one
checks `suspended` before emitting, with the reason written down: a second
`team.account.disabled` opens a second byte charge that only one `enabled` will
ever close. Deleting a team emitted unconditionally, so any partial failure
followed by a retry double-charged every seat the first attempt had already
disabled. The seat teardown moves into one method both paths can state the same
way, and a retry now resumes instead of re-charging.

Failure semantics are unchanged: a seat that throws still aborts the delete and
leaves the team live, which is what makes the retry meaningful.

Touching this file reformats a few untouched spans -- it carried ten
pre-existing `prettier/prettier` errors that the repo's own pre-commit `eslint
--fix` resolves on any commit that stages it.
2026-10-06 15:56:10 -04:00
Juan Castro 665314535e docs: say what TeamStore actually caches
The cache docblock claimed `isMember` and `getByUid` were left uncached "for
that reason". Both go through it -- `getByUid` directly, `isMember` through
`getMembership` -- so the comment told a reader the opposite of the code, on the
two reads that decide authorization.

What makes it safe is the busting, not an exemption: every mutation here drops
the keys it touches, and the TTL is the backstop for the one path that does not
come through this store, the ON DELETE CASCADE on the membership row. No stale
window is constructible today; the comment now says why.

Also drops a one-line docblock stranded above it, describing a method that is no
longer in the file.
2026-10-06 15:55:40 -04:00
Juan Castro f3b05ca822 fix: page GET /teams like every other team listing
`listTeamsForUser` returned every row and the controller never read `req.query`,
so `limit`, `cursor` and `includeTotal` were accepted and ignored on the one
team route that did not paginate. The docs promise a capped page there, and
every sibling listing already honours it.

Keyset-paginated on `id` per doc/pagination.md, with the envelope the other
routes return. The app-actor directory filter still runs over the page, so an
app sees the same teams it saw before, just a page at a time.
2026-10-06 15:55:22 -04:00
404oops 96f0c51d1e fix(ai): propagate cancellation and stream termination failures 2026-10-06 20:24:19 +02:00
Filip Kujundžić a1eca46a80 Merge pull request #4087 from HeyPuter/fix/responses-sampling-capabilities
fix(ai): omit unsupported sampling options in Responses requests
2026-10-06 20:22:16 +02:00
Daniel Salazar 484c498859 fix: error cleanup (#4089)
* fix: message

* fix: error cleanup
2026-10-06 11:16:04 -07:00
Reynaldi Chernando 71399c5939 sync limits ai, fs, events, networking (#4088) 2026-10-06 23:26:53 +07:00
404oops b1da48b738 fix(ai): omit unsupported Responses sampling options 2026-10-06 18:14:13 +02:00
Filip Kujundžić 9a9972b026 Merge pull request #4079 from HeyPuter/DS/danielsalazar/anthropic-route-fidelity
feat(ai): carry the full Anthropic and OpenAI wire shapes through the chat driver
2026-10-06 18:12:04 +02:00
Juan Castro 4629b544e6 docs: say why AppDriver.select lists apps it does not own
Reads as the same enumeration surface `/query/app` refuses to be, so write down
that it is not: `/query/app` answers for apps the caller names, which is why it
must not confirm one it may not see. `select` is the listing a launcher reads,
and a listing that hid every app but your own would not be one. Protected apps
are still filtered by the permission check, and `user-can-edit` narrows it to
the caller's own.

A test already pins the behaviour by name; this is the reason behind it.
2026-10-06 11:35:20 -04:00
Juan Castro cbb8d708b4 fix: scope the rewrite collapse to fs paths
The collapse caught every `subject_does_not_exist` the rewriters raise, not just
the path resolver's. `app-root-dir:` 404s when the caller's own app has no
resolvable root directory, and that answer is theirs to have — four of its tests
said so.

Only a permission that parses as an `fs:` path can oracle someone else's tree,
so only that one collapses now.
2026-10-06 11:33:27 -04:00
Juan Castro 95809c7dd5 fix: bound the permission check, and let whoami fail
`/auth/check-permissions` took an array of any length. `checkMany` batches the
scan so it is not the fan-out the report describes, but nothing stopped a caller
sending thousands of strings; it now takes the same 16 the grant and revoke
routes take.

`whoami()` returned `resp.json()` without looking at the status, so an expired
token resolved to an error body typed as a `User`. Callers read `username` off
it as undefined — which is how a consent prompt came to ask for
`/undefined/Documents`. It throws the documented `{ status, message }` shape
now, the same one the no-token path already threw.
2026-10-06 11:17:43 -04:00
Juan Castro ac5fa20388 fix: keep the other account out of the OIDC link conflict
Linking a provider identity already bound elsewhere answered 409 with the raw
provider subject and the other account's internal id in the message. The caller
needs to know there is a conflict, not whose it is; both stay in the log for
whoever is debugging it.

Status and legacy code are unchanged — OIDCService branches on the 409.
2026-10-06 11:14:19 -04:00
Juan Castro 3389d3f8a1 fix: treat an abuse hook that could not run as a refusal
The signup paths read `allow` off a payload they initialise to `true`, so a
harness listener that throws leaves the verdict untouched and the signup goes
through. The outer try/catch the report points at is not where it happens:
`emitAndWait` already swallows each listener's error and carries on, so that
catch almost never fires and the throw never reaches it.

`emitAndWait` now records the failure on the metadata it was handed, and both
signup paths — OIDC and the ordinary one — refuse when it is set. Other emitters
are unaffected: the flag is only read where somebody is asking for a verdict.

The report named OIDC alone; the ordinary signup had the same hole, and it is
the path that matters more.

Test boots a server, registers a listener that throws, and posts a signup: 200
before, 403 now.
2026-10-06 11:13:18 -04:00
Juan Castro 39099e2056 fix: size a shared folder from its owner, and keep sub-scopes when widening
Two unrelated bugs in the same area.

`stat({ returnSize: true })` summed `fsentries` filtered on the *acting* user,
so a recipient asking for the size of a folder shared with them matched no rows
and read 0. The sum is over the owner's rows; the ACL check that let them stat
it is what decides whether they may ask at all.

`fs-access-levels` destructured the mode out of the permission but kept the
`.slice(1)` that used to skip it, so widening `fs:<id>:see:thumbnail` produced
`fs:<id>:read` and dropped the sub-scope. Anything granted below the mode stopped
being implied by a wider one. Latent — nothing ships a sub-scope today — and the
test fails without the fix.
2026-10-06 11:05:52 -04:00
Juan Castro acd6ea3c75 fix: do not spend a consent prompt on an app root dir that is not the caller's
`perms.request('appRootDir', { app: <someone else's> })` could never succeed —
only the app itself may claim its root dir — but the SDK read the refusal as
"not granted yet", prompted the user, took the grant, then polled until it gave
up and resolved undefined. A prompt the user cannot act on, and a live grant for
a directory the app will never reach.

The route now says which kind of refusal it is. `forbidden` alone could not
carry it: the SDK already treats that as the refusal a grant is about to lift,
and a unit test pins exactly that sequence, so a new `reason: 'app_mismatch'`
field rides alongside instead of changing the code clients read.

An entry refused that way is held back from the pooled grant and resolves
undefined without a prompt, which is what it resolved to before.
2026-10-06 10:55:19 -04:00
Juan Castro d6c606fb21 fix: keep the owner's path out of the ghost-file 404
A share recipient addresses an entry as /<owner>/<uuid>/<name>, which is the
point of the mask: it says where the item is without naming the folders above
it. The 404 raised when a file's backing object has gone put `entry.path` in the
error fields, so reading or copying a shared file whose object had vanished
handed back the real path.

The fields carry the uuid alone now, on both the read and the copy site. The
caller already has it — it is what they named the entry by.

The existing ghost test asserted the path was present, so it is the regression
pin; the copy one was not checking fields at all and now does.
2026-10-06 10:33:59 -04:00
Juan Castro 8c7372adc6 fix: stop the path rewrite confirming entries the caller cannot see
`fs:/path:mode` resolves through `getEntryByPath` with no ACL check, and a
missing entry answers 404 while an existing one carries on to the authorization
check and answers 403. The pair is a membership test for any path on the
deployment: grant routes take the permission as given, so anyone could ask
whether /someone-else/taxes.pdf is there.

The resolver keeps its 404 for a path inside the caller's own home, where
knowing it is missing is the useful answer and reveals nothing. Everywhere else
it answers with the same `permission_denied` the authorization check gives, so
the two cases are one case. ACLService already collapses them; this is the same
rule reaching the grant and revoke paths, which take an actor and so can tell
whose tree is being named.

Internal resolution still goes through `rewritePermission`: only the paths that
take a caller-supplied permission moved over.

The test grants on an existing foreign path and a missing one and asserts the
outcomes are identical; without the fix they are 403 and 404.
2026-10-06 10:30:58 -04:00
Juan Fernando CastroandReynaldi Chernando 3711d1e837 docs: medium teams recipe (PUT-1976) (#3976)
* docs: add the medium teams recipe

Built around what an app actually does with a team: an assignee picker
keyed on uuid, paging a large directory, scoping stored data per team,
sharing with everyone at once, and telling the three empty cases apart.
One playground example per section.

* docs: split the teams recipe into the two use cases it covered

The medium teams recipe carried six sections, which is the wrong shape for a
recipe: an agent reading llms.txt to answer "add team sharing" had to find it
inside a page called "Work With a Team", next to paging and team switching.

Two recipes, one use case each:

  share-a-file-with-a-team   the whole team, or one teammate via the directory
  show-a-team-owner-panel    gating an admin view on isOwner

Dropped rather than moved: paging and the empty/closed-directory cases, both
already in Add Team Support, and per-team data namespacing, which assumes
multiple teams we do not support yet.

Two corrections while rewriting. A team share names the team in `holder_team`;
`holder` is a username, so the old example printed the wrong field and told the
reader it meant "no one person". And `share()` takes a username for a person,
never a uuid — the picker now shares by username and keeps the uuid for what it
stores, which is what the rename advice was always about.

The old page also linked to /recipes/teams-read-membership/, which does not
exist; the slug is add-team-support.

* refine

---------

Co-authored-by: Reynaldi Chernando <[email protected]>
2026-10-06 20:31:27 +07:00
Daniel Salazar 957784bdb8 fix(ai): keep existing chat callers on their previous behavior
Narrows the wire-shape changes to the callers that need them.

- request-level upstream 4xx stays strict only on the Anthropic route;
  /drivers/call and the OpenAI routes fall back on any error again
- outage chains return 400 with the old codes; the Anthropic error
  renderer maps them to 502/529 from the carried upstream status
- native Claude results report finish_reason 'stop' again (real reason
  in stopReason); OpenAI routes still map the real value, and a stream
  that carried tool calls never ends as a plain stop
- Claude usage carries thinking_tokens again, informational only and
  added after pricing and the funds cap
- reasoning_effort is clamped to values each OpenAI model accepts;
  Hoonify, MiniMax, ZAI and Meta only receive params they support
- OpenAI-form tool_choice from /drivers/call is normalized again
- /responses accepts background:false; mcp_servers is ignored; legacy
  media block shapes pass the Anthropic route's validation
- OpenAI web search holds one call; only search actions are billed;
  Azure bills Bing grounding requests; usd_cents includes search fees
2026-10-06 15:22:53 +02:00
Neal ShahandReynaldi Chernando dc029b43a8 Worker and Subdomain Recipes (#4075)
* Worker recipes medium

* subdomain recipes medium

* refine

---------

Co-authored-by: Reynaldi Chernando <[email protected]>
2026-10-06 20:09:38 +07:00
Daniel Salazar 985c76c9c3 docs: medium kv recipes (PUT-1972) (#4041)
* docs: medium kv recipes (PUT-1972)

Adds three recipes:
- Store a Set of Unique Values: an object as a set via update/remove,
  quoted paths for values with dots, like button, recent searches
- Run Code Only Once: incr as an atomic claim, once per day, quotas,
  sequential numbers, webhook dedupe in a worker
- Keep a Version History: incr version numbers, batch set with expireAt,
  reverse list paging, restore, keep last N, delete by prefix

* docs: more medium kv recipes (PUT-1972)

Adds six recipes:
- Keep an Index in Sync: batch set for record + index keys, moving keys
  on change, delete order, per-group counts vs includeTotal, search as
  you type, rebuild and stale-entry cleanup
- Share Data Between Apps: appData scopes, appUuid on every call,
  disableSharing private entries and the set-without-flag gotcha
- Save User Settings: defaults merge, nested paths, reset with remove,
  form binding, sets inside settings, renaming a setting
- Split a Large Value Across Keys: size-based read/write cost, shards
  numbered by incr, latest/older/all reads, expiring old shards
- Export or Delete Data: stream export, filtered batch import, delete by
  prefix, delete before a date, flush
- Build a Shopping Cart: quoted SKUs, incr quantities, TTL via update,
  hide-at-zero, server-side prices at checkout

Also adds a high score section to Add Counters, links Query a Collection
to the index recipe, and replaces its stale newest-first advice with
list({ reverse: true }).

* docs: simplify wording in medium kv recipes

Plainer prose throughout: shorter sentences, no bold or step-by-step
scaffolding, fewer defined terms. Code is unchanged apart from comments.
2026-10-06 19:10:27 +07:00
Daniel Salazar f21cd51edf fix(ai): retry a fast-mode 429 at standard speed on Claude
Fast mode has its own quota. Its 429 used to count as a route failure,
so the request was replayed on a reseller and the Claude route for that
model was marked unhealthy. The provider now retries once without speed.
2026-10-05 20:54:01 -07:00
Daniel Salazar c2fc45a8d1 fix(ai): keep the driver's token count off the /drivers/call surface
count_tokens is only for the Anthropic route, so the driver exposes it
under a symbol key that the RPC resolver never lists.
2026-10-05 16:56:48 -07:00
Juan Castro 614c4c9c2f test: pin the fs delegate path the escalation was reported on
The generic test covers the defect's shape; this one is the case the report
describes. `fs-access-levels` puts `manage:fs:<id>` in the higher set for
`fs:<id>:read`, so a delegate holding the manage terminates the issuer scan with
it as the leaf — which is how the old warm came to write a manage permission
under a holder granted nothing but read.

Without the fix the holder ends up with `manage:fs:<id>` in the flat view and
`canManagePermission` answers true for a read-only share, so they can re-share
the entry at any mode.
2026-10-05 18:19:25 -04:00
Juan Castro ac740984a3 fix: keep the flat permission view to the row it stands for
Two ways the flat KV view could say more than the SQL behind it.

On a miss the warm flattened the whole traversal and wrote every leaf under the
holder. Those leaves belong to the issuer: a holder granted one mode of a
resource by someone who holds the resource itself had the issuer's wider
permission written under their own key, and it answers later checks. The warm
now writes the holder's own row and nothing else, which is what the grant path
has always written.

Deleting an account cascaded the grants it issued out of SQL, but the flat
entries are keyed on the holder, so nothing removed them and they are trusted
without an issuer check. The access outlived the account, and the owner could no
longer see it to withdraw it. The refs are read before the row goes and cleared
after it, through the same batched delete a revoke uses, so peer regions hear
about it once.

Both fail without the fix: the warm test sees the issuer's wider permission
written under the holder, and the delete test reads the grant back from an
account that no longer exists.
2026-10-05 17:51:31 -04:00
Juan Fernando CastroandReynaldi Chernando 63eabf8c7c docs: medium fs recipe (PUT-1968) (#3975)
* docs: add the medium fs recipe

readdir paging, streaming, sorting and recursion, plus the write and
mkdir options that decide what happens to an existing name. One
playground example per section.

* docs: ground the medium fs recipe and examples in the UI that lists

A file browser with infinite scroll and a count footer, a gallery sort
header, an upload endpoint deciding what to do with a repeated filename,
and a report generator writing into a folder tree that does not exist.

* refine

---------

Co-authored-by: Reynaldi Chernando <[email protected]>
2026-10-05 21:38:10 +07:00
Juan Fernando CastroandReynaldi Chernando 445588f857 docs: medium perms recipe (PUT-1975) (#3974)
* docs: add the medium perms recipe

Batch requests behind one prompt, appData scopes, raw permission
strings with create, and the entries rejected before any dialog.
One playground example per section.

* docs: ground the medium perms recipe and examples in the screens that ask

A first-run setup, a settings page that renders its real state, a
companion widget reading another app's data, and an importer that wants
folders created on approval.

* refine

---------

Co-authored-by: Reynaldi Chernando <[email protected]>
2026-10-05 20:22:09 +07:00
Daniel Salazar 53ca2b61e5 fix(ai): review fixes for the wire-shape PR 2026-10-05 04:09:08 -07:00
Daniel Salazar a242e31343 feat(ai): carry the full Anthropic and OpenAI wire shapes through the chat driver
The Anthropic-compatible route flattened requests into the OpenAI-style
driver shape and rebuilt them for Claude, which dropped system blocks,
thinking, tool_choice, cache_control, betas and stop reasons. Claude
Code requests 400ed upstream and were silently served by a reseller.

- normalized driver args, stream chunks and results gain tool choice,
  thinking, output format, stop sequences, caching, stop reasons, usage
  details, reasoning signatures, server-tool blocks and safeguard results
- ClaudeProvider builds requests from those fields with a beta, tool and
  body-field allowlist; meters fast mode, web search and advisor calls,
  and no longer bills thinking tokens twice
- fallback only on route-level failures; outages map to 502/529
- vendor error envelopes on the Anthropic and OpenAI routes
- Anthropic route: native SSE with pings, count_tokens, /v1/models
- OpenAI-family providers and routes map the same fields; Responses web
  search is metered; reasoning_effort gating fixed; /responses rejects
  org-scoped stateful params and forces store:false
- puter.js: typedefs, new chat options, silent toString for new chunks
2026-10-05 03:35:54 -07:00
Daniel Salazar ba081e2332 feat(ddb): log the table and key of throttled DynamoDB requests (#4078)
A client middleware inside the SDK retry loop logs every throttled attempt
(including ones that succeed on retry) with its operation, table, key,
client region and DynamoDB's throttling reason, plus batch items returned
unprocessed. Each target logs at most once a minute with a count of the
throttles folded into it. Puts log only the key attributes, using key
schemas registered by the KV store at startup.
2026-10-05 02:01:38 -07:00
Juan Fernando CastroandDaniel Salazar a46eb15d5e Let the account's own API token reach the routes a privileged app needs (#4032)
* feat: let the account's own API token reach the routes it should

A privileged app is launched with the GUI session token today. Running one on a
full-access personal access token instead costs much less when it leaks, but
several routes those apps depend on refuse every access token, so the swap would
break them.

Admitted, each a read or a write the token's own HTTP reach already covers:

  - the realtime socket handshake, which refused access tokens outright; FS
    live updates and notifications stop without it
  - GET /get-dev-profile and POST /profile, both puter.js surface
  - GET /auth/list-permissions — a read of what was granted, not a grant
  - GET /share/shared-by-me/apps and GET /share/audit, over grants
    /share/shared-by-me already lists for the same credential
  - POST /rao, which puter.js calls on every setAuthToken

Gated on `isAccountContext`, not the `full_access` claim alone, so a token with
an app anywhere in its chain is still refused and a socket for one never joins
the user room.

Left session-only: mint, session, 2FA, grant, team and billing, plus two the
list named. /share/blocks is a personal safety control whose only caller is the
desktop's Blocked Senders window. /app-feedback is reachable only from our own
GUI pages, and being unsubmittable programmatically on a user's behalf is the
property it was built with.

The new HTTP suite drives a real server with a minted token and fails on all six
admissions without the change; the block routes' existing metadata check already
pins their refusal.

* test: sort the two token shapes at the events handshake

* fix: carry the account-context invariant into the HTTP gate

`requireUserActorGate` admitted on the raw `full_access` claim while the socket
and /rao use `isAccountContext`, so one shape — a token carrying the claim with
an app in its chain, or an actor built without `makeActor` — would have been
refused in two places and admitted in the third. Unreachable today, since the
mint and the auth-time read both drop the claim when `app_uid` is present, but
it is the invariant this change states and it should hold at every enforcement
point.

The payout address goes back behind a plain session: `/get-dev-profile` is open
to the account's own token now, and the rest of what it answers is a developer's
own name and incentive-program status.

Also drops a cross-reference in EventsService to the socket posture this change
relaxed, and rewords the block-routes comment, which read as if access tokens
were admitted there rather than refused.

* fix: drop a revoked token's sockets, and read the dev-profile columns

Two gaps the review found, both reachable only because this branch lets the
account's own token hold a socket and call /get-dev-profile.

`#revokeAccessTokenTail` soft-revoked the session row and told no one, so a
revoked token kept its connection — and the account's `outer.gui.*` fan with it
— until the five-minute reauth sweep. Sockets now join a per-token room and a
new `auth.access-token.revoked` event drops exactly that room, so the account's
other tabs stay up. Session revoke keeps its account-wide eviction.

`/get-dev-profile` read `first_name`, `last_name`, `paypal` and the two
incentive flags off the user row, which carries none of them: the columns have
always been `dev_`-prefixed, so the endpoint answered nulls to everyone and
`puter.apps.getDeveloperProfile()` has never returned anything. Reads the real
columns, keeping the unprefixed name as a fallback for a deployment carrying
both. The payout address stays behind a plain session, so opening the route to
the account's token hands over a name and incentive status and nothing else.

Both fail without the fix: the socket test waits five seconds on a connection
that should already be gone, and the profile test reads back a seeded row.

* fix: close /open_item to tokens, and stop the node runner leaking sockets

`/open_item` carried `allowFullAccessToken`, but it grants an app a permission
on the user's behalf and mints the app token to go with it — a grant and a mint,
which stay session-only. Pre-existing, and missed by the first sweep here, which
only asked whether routes lacking the flag should gain it and never re-read the
ones that already had it. Refused at the top of the handler too, so a credential
that gets past the gate cannot leave an ACL row behind.

The node runner loads each SDK into a vm context and never closed it, so every
test leaked a socket, and every socket holds a per-(user, origin) connection
slot. Admitting the account's own token to the handshake pushed that past the
cap, and the whole events block failed with `events_connection_failed` while the
browser and workerd runners — which do not leak — passed. Closing the FS socket
and the events channel after each test gives the slots back.

* fix: announce only the revokes that can have a socket to drop

Every access-token revoke broadcast an eviction cluster-wide, including the
read-URL tokens `revokeReadUrl` retires constantly. A scoped token is refused at
the handshake, so none of those broadcasts could ever reach a connection.

The announcement now goes out only for a token the handshake would have
admitted: full access, no app in the chain. `revokeOwnAccessToken` refuses
full-access tokens outright, so that whole path is silent; a revoke by raw uuid
says nothing about the token and still announces, which costs a no-op broadcast
rather than leaving a revoked socket up.

Fails without it: the test revokes a scoped token and a full-access one through
the same entry point and counts the announcements.

* test: pin the mint route against the account's own token

`/auth/create-access-token` carries only `requireAuth`, so it never showed up
in the sweep over `requireUserActor` routes — the refusal lives in
`AuthService.createAccessToken`, which turns away any access-token actor
outright. That holds today, and a leaked full-access token minting itself a
sibling that survives revoking the original is exactly what the token swap is
meant to rule out, so it is worth a test rather than a reading.

* fix: let the recommended-apps test follow the list it tests

Two commits on main rewrote `RECOMMENDED_APP_NAMES` without touching the test,
which seeds `editor` and `camera` and asserts the result contains them. Neither
is on the list any more, so the resolved set came back empty and `test (base)`
has failed on every PR since.

The test now seeds the first two names off the list itself and asserts on those,
so the next edit to it cannot strand the test: prepending two unknown apps keeps
all four cases green.

* feat: let an extension waive a plan gate for one route or driver method

A plan gate about to refuse a caller now emits
`subscription.gate.<surface>` through `emitAndWait` first:
`route.<method>.<path>` for a route, `driver.<iface>.<method>` for a driver
method, `share.anyone` for link shares. The payload carries the actor, the
request and the requirement, and a listener that sets `allow` waives the plan
for that call. Callers the plan already covers never reach the hook, and a
listener that throws leaves the refusal in place.

* feat: mint godmode apps their own full-access token

`/auth/get-user-app-token` answers a desktop launch of a godmode app (the
`godmode` column) with a full-access token instead of an app token. Origin
lookups, the sign-in popups for pages outside the desktop, keep getting
ordinary app tokens, and only a web session may mint one.

The token carries the app in `godmode_app_uid`, read into
`accessToken.godmodeApp` for attribution only: `effectiveApp` stays null, so
it keeps the account's reach and passes the same gates as a personal access
token, including the vendor-compatible AI routes. It is refused once the app
is gone or no longer godmode.

Its row is an `access_token` session parented to the web session that asked,
with `app_uid` set, so signing out, revoking that session or revoking all
sessions takes it along, and the sessions list shows it under the app. Each
mint re-signs onto the same row with a 12h `exp` and moves the row's expiry
out, so the desktop keeps an open app alive while a copy taken earlier still
expires on schedule.

A full-access token may now mint scoped tokens (read URLs), parented to its
own row; it still may not mint full access. Revoking a full-access token
cascades to the tokens it minted.

An access token whose session row has a parent now authenticates only while
that parent is live, so a read URL a godmode app minted lapses with the app's
token, and through it with the desktop session.

* feat: launch godmode apps on their own token and renew it

The desktop asks for a godmode app's token at launch and blocks the launch if
it can't get one, as it does for other apps; a godmode app never runs on the
desktop's session token. While the window is open the desktop re-mints the
token once less than 6h of its 12h remain, checking on a timer and when the
tab becomes visible, and answers the app's `reauth_required` with a fresh one,
posted to the app's origin.

puter.js keeps a godmode token in sessionStorage like a session token, never
in localStorage. In app mode a 401 on a godmode token, background requests
included, asks the desktop for a new token and replays, and a `puter.token`
message is only taken from the embedding frame.

The sessions list shows a godmode token row with its app's title and icon.

* fix: keep a godmode token the same across renewals

Apps hand their launch token to code that never sees a replacement (the
terminal's Drive mount, a CLI's API key), so a token re-signed every renewal
broke those copies once the first one expired. The JWT now carries no `exp`
and a renewal moves only its row's expiry, so every copy keeps working while
the desktop keeps the app alive.

What still bounds it:
- only the desktop's web session can extend it; using the token never does,
  and signing out or revoking that session revokes it;
- it lapses 12h after the desktop last asked, and a lapsed row is never
  revived: the next launch gets a new token;
- however often it is renewed, a token is replaced after 7 days, and the old
  one is revoked on the spot along with the tokens it minted and its sockets.

The desktop no longer re-posts a renewal that kept the same token, so the
app's sockets aren't rebuilt every few hours; an app that reports its token
stopped working is always answered.

* fix: reserve the feedback and no-reply mailbox names again

Signup checks names against `util/reservedUsernames`, but `fbl`, `noreply` and
`no-reply` were added to a second copy of the list in AuthController that
nothing reads any more, so they became claimable. Move them into the live list
and drop the copy.

---------

Co-authored-by: Daniel Salazar <[email protected]>
2026-10-05 01:10:29 -07:00
Neal Shah 970092f0a0 PUT-1995 (#4020) 2026-10-05 03:26:54 -04:00
Neal Shah 6dc61b2d67 PUT-2011 (#4077) 2026-10-05 03:23:14 -04:00
Neal Shah 7976dc9a29 attach egress actor to /down (#4076) 2026-10-05 03:09:16 -04:00
Daniel Salazar a76800d693 fix(thumbnails): duplicate a copied thumbnail with GetObject + PutObject instead of CopyObject (#4072)
* fix(thumbnails): duplicate a copied thumbnail with GetObject + PutObject instead of CopyObject

CopyObject names its source as `<bucket>/<key>` in a header, which skips
any path in the client's endpoint, so it can miss an object that
Bucket/Key requests reach. The copy handler now reads the source object
(still behind the HeadObject size bound) and writes it under the copy's
key with its content type.

* fix(thumbnails): check the GetObject length before buffering a copied thumbnail

The size check runs on a HEAD and the read on a separate GET, so an object
replaced in between was read whole into memory. Reject a GET response whose
length exceeds the cap and destroy its body before reading.
2026-10-04 21:11:00 -07:00
Daniel Salazar 1f1e5674d8 chore: bump puter js ver (#4073) 2026-10-04 17:17:38 -07:00
Daniel Salazar 8db4fb4f5b fix(ai): chat credit gate ceiling, in-flight holds and disconnects (#4071)
* docs(agents): no credits or bug-report mentions in commits and PRs

* fix(ai): chat credit gate ceiling, in-flight holds and disconnects

- Cap output at min(max_tokens, context - prompt). The gate used
  max_tokens - prompt, so any prompt longer than a model's output limit
  was refused with 402 regardless of balance. Gemini 2.5 Pro's output
  limit corrected to 65,536.
- Abort the upstream stream when the caller disconnects (driver calls and
  the OpenAI/Anthropic-compatible routes), which ends the credit hold.
- A request refused only because the account's own in-flight requests
  hold its balance gets 429 `credits_reserved` instead of 402; puter.js
  retries it with exponential backoff (1s to 30s, about 2 minutes).
- When the balance caps a completion and it uses the whole cap, the
  response carries `usage_limited` (non-stream `metadata`, stream `usage`
  line), which puter.js already turns into the upgrade prompt.

* perf(ai): read the balance and its holds once per credit gate

The gate re-read holds, and the whole balance when any were held, to tell
spent from committed. getUsageHeadroom returns both halves from the same
reads getRemainingUsage already did.
2026-10-04 17:10:51 -07:00
Daniel Salazar e7c89b0639 fix(apps): keep ids longer than an app uid out of the app-stats query (#4070)
ClickHouse's app_opens.app_uid is FixedString(40) and fails the whole
query on a longer value. Marketplace callers pass item ids into
getAppsStats, so one long listing-request id turned the seller's
listing-request list into a 500. Such ids can't have opens; they now
resolve to zero without being queried.
2026-10-04 17:03:19 -07:00
Daniel Salazar 83d64ac685 fix: follow-ups from the reviews of #4047, #4049, #4051, #4053, #4055 and #4057 (#4068)
* fix(auth): issue reauth tokens only for browser sessions

A rejected token's 401 carried a signed reauth_token, which /signup turns
into a session on a temp account. Any revoked app or access token, a stale
token from a deleted app whose uid was reused, or a worker credential could
get one. Only a GUI/browser session's rejection now carries a reauth_token;
everything else still gets reauth_required without it. The stale-app check
also runs before the session is touched.

* fix(events): refuse worker creation from an events handler

workers.create minted a worker token with no handler depth, so a handler
could deploy a worker and chain writes through it at depth 0. A handler
actor can no longer create a worker; delivery, deploys and hot reload mint
from plain owner actors and are unaffected.

* fix(ai-ocr): bound the page count's parsing and close its under-counts

- Object-stream headers are capped (/N, /First) and read incrementally, and
  the inflate budget scales with the file, so a crafted PDF can't spend
  seconds and gigabytes before the credit check.
- An escaped /ObjStm type, an indirect /Count, untyped page leaves and an
  object redefined to a smaller tree no longer lower the count.
- An account with no balance is refused before the file is parsed.

* fix(fs): add a switch to stop accepting pre-binding signed URLs

Signed URLs issued before owner binding are still accepted, and owner-signed
ones never expire, so the move-to-another-tree attack stays open for them
indefinitely. legacy_file_signatures (default on) can now turn them off, a
rate-limited warning counts their use so it's visible when they've died
out, and a non-UUID uid is refused before the entry lookup.

* fix(fs): hide the issuer's home uid from a scoped token's entries

A token scoped below the home could still learn the home's uid as the
parent_uid of a direct child (e.g. stat on ~/Documents). For access-token
actors that can't list the home, a direct child's parent uid is now null in
the v2 and legacy entry shapes, using the same rule as the root listing.

* fix(gui): ask for the 2FA code on the last step, right before enabling

The setup flow verified a code before the recovery-codes screen and sent
that same code to enable afterwards; by then it had usually expired, and
the user was sent back to the start. The code is now entered last and goes
straight to enable; a wrong code keeps the user on that step, and other
failures show the server's message. Enter submits the password step.

* test(fs): expect the legacy-signature flag in the signing config

* fix: apply the review's follow-ups to the merged-PR fixes

- Reauth tokens also require the session row to be a browser ('web')
  session, not just the absence of the worker claim.
- The OCR page count only treats an untyped object as a page when it sits in
  a page tree's /Kids, so outline items, widgets and name-tree leaves can't
  inflate an honest PDF's count.
- The signed-URL uid guard checks the uuid's shape only, so a legacy row
  whose uuid has nonstandard version/variant bits still resolves.
2026-10-04 14:46:21 -07:00
Daniel Salazar 5e34258da5 fix: drop database_id from the subdomain reclaim; MySQL has no such column (#4067)
reclaimByUuid (#4064) cleared `database_id`, which exists only in the SQLite
and Postgres schemas. On MySQL the UPDATE fails with "Unknown column", so
ensureSystemSite throws: the puter-profiles row stays registered to the
admin, the site keeps 404ing, and every profile save returns 500 as it
retries the bootstrap. puter-app-icons takes the same path if its row isn't
already protected. System rows never carry a database binding, so there is
nothing to clear.
2026-10-04 02:45:03 -07:00
Daniel Salazar 51eba174b3 fix: alert errors from timeouts (#4066) 2026-10-04 02:44:40 -07:00
Daniel Salazar f533eb84ae fix(ai-ocr): check credits for the whole document before calling the provider (#4055)
* fix(ai-ocr): check credits for the whole document before calling the provider

The pre-flight checked one page's cost because the page count was only
known from the provider's answer, so a 130-page PDF passed on a balance
worth a fraction of it and drove the account negative.

The driver now estimates pages before the call: a PDF's own count (read
from its page tree, including compressed object streams), capped by a
`pages` selection; one page for images and Textract, whose synchronous
API reads single-page documents. Other documents and PDFs it can't read
count 20 pages per MB, Mistral's 1,000-page limit at its 50 MB size cap.
The estimated cost is checked and held while the provider runs, so
concurrent calls see it. Usage is still metered from the pages the
provider reports.

* fix(ai-ocr): price the credit hold at the AI cost factor

Billing scales OCR usage by the ai.cost.factor hook, but the hold was sized
at the raw cost, so a balance covering N pages passed the check and was
then charged more. Hold through reserveAiCredits like the other AI drivers.
2026-10-04 00:53:19 -07:00
Nikhil Jangid 3dfc70cab7 fix: clarify team directory app access (#4031) 2026-10-04 00:51:02 -07:00
Daniel Salazar ebf5086e1c fix(fs): keep restore metadata out of reach of anyone but the tree's owner (PUT-2050) (#4063)
Follow-up to #4048. original_path/original_name decide where a GUI restore
lands, and the owner's restore is unchecked, so whoever sets them can aim a
file at another app's AppData root and block that app's launch. #4048 only
pinned them on move for non-account actors; a write could still set them,
and a share recipient's session counts as account context.

Writes now drop the trash keys for every caller. On move, only the account
that owns both source and destination may set them; anyone else moving into
the owner's Trash gets them recomputed from the entry, and other moves drop
them, including keys already on the row. Copies drop them on every branch.
The GUI only offers Restore for direct children of Trash, since nested rows
never carry legitimate trash metadata. The AppData/Trash checks for
cross-app deletes and create grants now ignore case, like the root guard.
2026-10-04 00:48:48 -07:00
Daniel Salazar 59e28bed23 fix(metering): keep a failed lookup's fallback out of the credit cache (PUT-2002) (#4062)
Follow-up to #4052. A failed subscription lookup was cached for 5 s, but the
credit cache built from its fallback plan lived 15 s, so a paid user who
had spent past the free allowance got 402s for ~15 s after one failed read.
Credit entries built from a provisional answer (a failed lookup, or an
actor missing its email or numeric id) are now capped at that answer's
expiry and don't drive credit-state alerts.

A failed lookup that finishes after a concurrent successful one no longer
replaces the good cached answer, actors without a numeric id aren't cached
(resolvers key on it), and a failed credit refresh fails open briefly
instead of replaying a stale answer or blocking on the failing store.
2026-10-04 00:48:35 -07:00
Daniel Salazar 73ebe3f47a fix(metering): decide app-usage access in MeteringService only (PUT-2015) (#4061)
The /metering/usage/:app handler repeated the "an app reads only its own
usage" check while the service still let any app read the global bucket.
The service now refuses an app actor anything but its own app, and the
handler only resolves names. Status codes for app actors are unchanged; a
user reading os-global gets their global usage instead of a name-lookup 404.

whoami no longer sets taskbar_items to undefined for app actors, and only
keys with a value count as built, so a listener can't fill one in for an
app. getMonthlyUsage's JSDoc notes allowanceInfo is account-wide.
2026-10-04 00:47:37 -07:00
Daniel Salazar c981a51f21 fix(fs): bind signed file URLs to the owner they were issued under (#4053)
New /sign signatures are an HMAC over uid, action, expiry and the entry's
owner at signing time. /file and /writeFile resolve the entry first and
verify against its current owner, so a move that hands the entry to another
account ends its URLs and stops /file minting child URLs from them. The URL
shape is unchanged; signatures in the previous format still verify, without
the owner check.
2026-10-04 00:47:16 -07:00
Daniel Salazar 4ef5e6c110 fix(socket): hold unlisted free plans to the free connection cap (PUT-1830) (#4060)
The per-account socket cap looked a plan up directly in its map, so
org_seat_free (team seats on a free team) got the paid cap. Route and
events limits already gave an unlisted free plan the default free entry;
that rule now lives in one helper, subscriptionOverride, used by all three.

Also publishes the socket connection caps, corrects the seat-plan line in
rate-limits-and-quotas.md, and makes the isFreeSubscription test assert
literal ids.
2026-10-04 00:46:59 -07:00
Daniel Salazar 794469845d fix(thumbnails): bind thumbnail keys to their entry and size-bound uploads (PUT-1999) (#4054) 2026-10-04 00:46:35 -07:00
Daniel Salazar 9ca77be5a9 fix(auth): require a step-up to set up 2FA and a valid code to enable it (#4051)
Setup returned a fresh secret and recovery codes to any session, and enable
flipped otp_enabled without proof of the secret, so a stolen session could
enroll its own authenticator and lock the owner out. Setup now lives at
/user-protected/setup-2fa behind the same password / OIDC revalidation gate as
disable-2fa, and enable requires a live code for the stored secret. The GUI
setup window asks for the password (or opens the revalidation popup) first,
and sends the code it already collected to enable.
2026-10-04 00:26:28 -07:00
Daniel Salazar 007d5391a1 fix(events): carry handler depth onto tokens a handler mints (#4049)
An events handler could mint a fresh token for its own app through /sign or
/auth/create-access-token, and writes made with it started a new chain. An app
token minted from a handler token now carries its handler_depth and expires
with it; an access token carries the depth at the lifetime asked for.
2026-10-04 00:25:54 -07:00
Daniel Salazar c0f4fe93b1 fix(auth): stop a deleted app's sessions authenticating as the next app with its uid (PUT-2049) (#4057) 2026-10-04 00:24:55 -07:00
Daniel Salazar d57f638eda fix: keep system sites writable and their names out of reach (#4064)
Follow-up to #4056. Icons and profiles are now written as the system user,
so on a storage-limited install they counted against its allowance (100 MB
by default) and every write past that returned 413, including a user's
profile save. Both writes now pass the unmetered allowance, as writes the
system makes on users' behalf already may.

ensureSystemSite reclaims a mismatched subdomain row in place instead of
delete+create, so the name is never free for a user to register, the row
keeps its uuid, and `protected` is never cleared; the previous holder's
bindings are dropped. If the row vanished since the read it falls through
to create. puter-app-icons is now protected like puter-profiles, both names
are reserved from user registration, and they live in util/systemSite so
the subdomain driver doesn't import the services. The icon service's
shutdown wait is capped at 5 s so a stalled run can't hold up the metering
flush.
2026-10-04 00:23:01 -07:00
Daniel Salazar 26abd4e4a5 fix(fs): hide the issuer's home from scoped-token root listings (PUT-2053) (#4047) 2026-10-04 00:22:43 -07:00
Daniel Salazar 3defae3cfc fix: own system sites by the system user (#4056)
* fix: register system sites to the owner of their directory

The hosting middleware refuses a site whose user doesn't own its root
directory (absent a manage grant). The app-icons and profiles bootstraps
always registered their subdomain to the admin, but the directory can belong
to another user: a carried-over `/system` owned by the `system` user, which a
new child directory inherits. Those sites then 404 on every request.

Both bootstraps now go through ensureSystemSite, which registers the
subdomain to the directory's owner and replaces a row registered to anyone
else or pointing at another directory.

Fixes #4045

* fix: own system sites by the system user

`/system`, `/system/app_icons` and `/system/profiles` and their subdomains
belong to the system user, and icon and profile files are written as it.
ensureSystemSite re-owns directories another account created and replaces a
subdomain row registered to anyone else, so installs bootstrapped under the
admin move over on boot.

The bootstrap no longer waits for an admin account, so DefaultUserService
stops re-running it, and deployments without an admin get icons and profiles.
AppIconService lets in-flight icon runs finish on shutdown.
2026-10-03 17:06:13 -07:00
Daniel Salazar 205f19a167 fix(metering): stop caching subscription answers from email-less actors and failed lookups (PUT-2002) (#4052) 2026-10-03 16:39:08 -07:00
Daniel Salazar 2cfc1461e4 fix: scope whoami billing details and monthly usage to the calling app (PUT-2015) (#4050)
* fix: scope whoami billing details and monthly usage to the calling app (PUT-2015)

* fix(metering): refuse an app another app's detailed usage (PUT-2015)
2026-10-03 16:38:31 -07:00
Daniel Salazar 3fa23bb809 fix(fs): refuse renaming or moving AppData roots (PUT-2050) (#4048)
* fix(fs): refuse renaming or moving AppData roots (PUT-2050)

* fix(fs): refuse a forged trash original_path from apps (PUT-2050)
2026-10-03 16:37:36 -07:00
Daniel Salazar e59f6285ba refactor(metering): route free-plan checks through isFreeSubscription (PUT-1830) (#4046) 2026-10-03 16:36:05 -07:00
Filip Kujundžić c3abec8663 Merge pull request #4035 from HeyPuter/filipkujundzic/put-1895-chat-credit-gate-cost-factor
fix(ai): price the chat credit gate at the factored cost (PUT-1895)
2026-10-03 19:44:51 +02:00
Neal Shah 04406373f4 event based FBL implementation (#3969) 2026-10-03 05:54:04 -04:00
Daniel Salazar 91aff909e1 fix: keep launch tokens out of URLs and storage (PUT-2040) (#4038)
* fix: keep launch tokens out of URLs and storage (PUT-2040)

- Strip the consumed launch token param from the app URL after load
- Never persist user session (godmode) tokens to localStorage; purge any stored one at boot
- Worker log socket sends the token as its first message instead of in the URL path

* fix: strip the private-app sign-in token from top-level URLs, keep godmode tokens across reloads (PUT-2040)

- Private-app gate: a top-level GET carrying ?puter.auth.token= that passes the gate gets the sticky cookie and a 302 to the same URL without the token. Framed launches are untouched.
- Godmode session tokens go to sessionStorage (never localStorage), so a frame reload stays signed in; cleared on sign-out or when an app token replaces it.
2026-10-03 00:26:31 -07:00
Daniel Salazar a242105540 fix(events): keep app-less scoped tokens off the account's durable rows and mailbox (#4042)
A personal access token scoped below the account (e.g. the one in an
account-minted getReadURL() URL) has effectiveApp === null, which the durable
list/unsubscribe/ack scope check and notif fetch read as account context. Gate
those on isAccountContext, as workers/handlers/kv-handles already do, and refuse
such tokens on durable subscribe.
2026-10-02 22:37:07 -07:00
Reynaldi Chernando 47510897ed add ai builder link to docs (#4044) 2026-10-03 10:27:23 +07:00
dependabot[bot] d0da8c5b1e chore(deps): bump engine.io from 6.6.9 to 6.6.11 (#4017)
Bumps [engine.io](https://github.com/socketio/socket.io) from 6.6.9 to 6.6.11.
- [Release notes](https://github.com/socketio/socket.io/releases)
- [Changelog](https://github.com/socketio/socket.io/blob/main/CHANGELOG.md)
- [Commits](https://github.com/socketio/socket.io/compare/[email protected]@6.6.11)

---
updated-dependencies:
- dependency-name: engine.io
  dependency-version: 6.6.11
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-02 15:03:34 -07:00
dependabot[bot] 303ee6c2c7 chore(deps-dev): bump brace-expansion from 1.1.18 to 1.1.21 (#4015)
Bumps [brace-expansion](https://github.com/juliangruber/brace-expansion) from 1.1.18 to 1.1.21.
- [Release notes](https://github.com/juliangruber/brace-expansion/releases)
- [Commits](https://github.com/juliangruber/brace-expansion/compare/v1.1.18...v1.1.21)

---
updated-dependencies:
- dependency-name: brace-expansion
  dependency-version: 1.1.21
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-02 15:02:56 -07:00
Daniel Salazar 37c1678897 fix(fs): keep a vacated username off-limits to other accounts for an hour (#4039)
* fix(fs): keep a vacated username off-limits to other accounts for an hour

Renaming a home rewrites descendant paths in the database, but their
cached entries keep the old path until they expire. ACL grants a home by
path prefix, so whoever takes the old name next must not be able to
before those entries are gone. renameUserHome now records the vacated
name, and the claim check every username-claim site already runs treats
it as taken for anyone but the account that left it.

* test: seed recommended apps that are still in the default list

The default list no longer includes editor or camera, so the ordering
test found neither.
2026-10-02 14:53:34 -07:00
Filip KujundžićandClaude Opus 5.5 7b8b3752e4 fix(ai): price the chat credit gate at the factored cost
The chat gate read `model.costs` directly to decide affordability, cap
`max_tokens` and size the credit hold, while usage is recorded at cost
times the AI cost factor, so all three ran low by the factor on every
request. The gate now resolves the factor and applies it to both rates.

The factor is looked up by the key the provider records usage under.
Providers expose that key through `meteringModelKey` and use it when
recording, so the gate and the charge can't drift apart; the
unreported-stream backstop records under the same key.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-02 21:21:18 +02:00
404oopsandClaude Opus 5.5 521812be02 refactor(ai): move the AI cost factor out of MeteringService
MeteringService knew about AI: it built the cost-factor facade, resolved
the `ai.cost.factor.*` hook and, since the previous commit, offered
`reserveAiCredits`. Metering stays agnostic to what it meters, so all of
that now lives with the AI drivers in `drivers/util/aiCostFactor.ts`.

`withAiCostFactor(metering, events, driver)` returns an
`AiMeteringService`: the service with recording scaled by the factor,
plus `costFactor` and `reserveAiCredits` for gates. The speech providers
take that type. MeteringService itself has no AI methods left.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
2026-10-02 21:20:24 +02:00
Nariman Jelveh bb38e8c86f Update RecommendedAppsService.ts 2026-10-02 11:29:42 -07:00
Nariman Jelveh 74d89b64fd Update RecommendedAppsService.ts 2026-10-02 11:19:37 -07:00
Daniel Salazar 31e5b64a0f fix: make app-data delete grants imply write and read (PUT-1998) (#4022) 2026-10-02 10:47:58 -07:00
Daniel Salazar e96cf77982 feat(metering): per-month allowance override for partly billed plan changes (PUT-1996) (#4027) 2026-10-02 10:40:52 -07:00
Filip KujundžićandClaude Opus 5.5 22d52cf544 fix(ai): name data-URL audio with its real extension
A data URL's filename came from its MIME subtype, so `audio/mpeg`
became `input.mpeg` and `audio/x-wav` became `input.x-wav`. OpenAI's
transcription API picks the decoder from the extension and rejected
both. Map the MIME types whose subtype isn't the extension.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-02 19:28:13 +02:00
Daniel Salazar 86bb94d3d9 fix: cap FS batch array lengths (PUT-1894) (#4024) 2026-10-02 10:06:21 -07:00
Daniel Salazar a4f1e665df fix: return 400 when a streamed upload is shorter than its declared size (#4023)
A streamed write sends the caller's declared size to the object store as
the body length. A body that ends short of it was rejected as
IncompleteBody and surfaced as a 500.
2026-10-02 10:05:13 -07:00
Filip KujundžićandClaude Opus 5.5 abd0d809d5 fix(ai): price speech credit gates at the charged cost and hold them in flight
Text-to-speech, speech-to-text and voice-changer providers checked
affordability against the unscaled provider cost and never reserved it,
so the gate ran low by the AI cost factor and concurrent requests could
each spend the whole balance. They now go through
`reserveAiCredits`, which scales the amount by the model's cost factor,
checks it, and holds it until the request finishes.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-02 18:56:53 +02:00
Filip KujundžićandClaude Opus 5.5 82bd7a7360 chore(ai): sync model catalogs with provider listings and deprecations
Remove models providers have deprecated or retired, add newly released
ones, and move provider defaults off retired models. Gemini video (Veo
via the Gemini API) is removed; Infron and OpenRouter skip models their
listings mark deprecated. xAI bills cached prompt tokens only at the
cached rate. Claude Sonnet 4.6 max output raised to 128K.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-02 18:52:35 +02:00
Reynaldi Chernando b32ae7dcfc Add rate limit detail for contact form recipe (#4030)
* Add rate limit detail for contact form recipe

* handle ipv6
2026-10-02 23:19:41 +07:00
Juan Fernando Castro d5e6d5ea07 Merge pull request #4007 from HeyPuter/juancastro/put-1896-file-and-sandboxed-iframe-origins-get-a-hard-400-and-cant
fix: refuse opaque origins cleanly instead of a logged 400 (PUT-1896)
2026-10-02 09:58:13 -04:00
Reynaldi ChernandoandCopilot Autofix powered by AI f46d187d51 contact form recipe (#4029)
* contact form recipe

* Potential fix for pull request finding 'Network failures cause unhandled fetch rejections'

Co-authored-by: Copilot Autofix powered by AI <[email protected]>

* minor

---------

Co-authored-by: Copilot Autofix powered by AI <[email protected]>
2026-10-02 20:24:05 +07:00
Reynaldi ChernandoandCopilot Autofix powered by AI 0fcee62884 Update recipes for storing a small list and store per id (#4028)
* Update recipes for storing a small list and store per id

* Potential fix for pull request finding 'Clarify that IDs must be path-safe or properly escaped'

Co-authored-by: Copilot Autofix powered by AI <[email protected]>

---------

Co-authored-by: Copilot Autofix powered by AI <[email protected]>
2026-10-02 19:23:39 +07:00
Reynaldi Chernando 384524a027 improve teams docs (#4025) 2026-10-02 14:45:54 +07:00
Daniel Salazar 42b204f46e fix: don't count failed username change towards rate limit (#4021) 2026-10-02 00:26:37 -07:00
Daniel Salazar 91c8b2385b fix: run a broadcast persistent handler in the worker or the clients, not both (PUT-1889) (#4016) 2026-10-02 00:08:02 -07:00
jelveh 05e2e7428a feat(gui): let users remove their profile picture
The account tab only offered changing the avatar. Add a "Remove photo"
action, shown while a picture is set, that clears it via
update_profile({ picture: null }) and resets every avatar to the default.
On failure the hint line says so and the picture stays.

The tab can render before the profile loads, so the profile loader also
reveals the button once a picture arrives. Also move the avatar hint
into i18n.
2026-10-01 23:21:57 -07:00
dependabot[bot] f387a7aa68 chore(deps): bump fast-uri from 3.1.7 to 3.1.8 (#4014)
Bumps [fast-uri](https://github.com/fastify/fast-uri) from 3.1.7 to 3.1.8.
- [Release notes](https://github.com/fastify/fast-uri/releases)
- [Commits](https://github.com/fastify/fast-uri/compare/v3.1.7...v3.1.8)

---
updated-dependencies:
- dependency-name: fast-uri
  dependency-version: 3.1.8
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-01 18:39:45 -07:00
Daniel Salazar e6bda950f0 perf: batch per-app origin, canonical and hosted-backing lookups in app listings (PUT-1892) (#4013) 2026-10-01 18:30:06 -07:00
Daniel Salazar 4f31388dad fix: cap app lookups and fields, dedupe icon processing, rate-limit app landing pages (PUT-1891) (#4012) 2026-10-01 17:54:25 -07:00
dependabot[bot] 376a047ecc chore(deps): bump @grpc/grpc-js from 1.14.4 to 1.14.5 (#4010)
Bumps [@grpc/grpc-js](https://github.com/grpc/grpc-node) from 1.14.4 to 1.14.5.
- [Release notes](https://github.com/grpc/grpc-node/releases)
- [Commits](https://github.com/grpc/grpc-node/compare/@grpc/[email protected]...@grpc/[email protected])

---
updated-dependencies:
- dependency-name: "@grpc/grpc-js"
  dependency-version: 1.14.5
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-01 17:50:14 -07:00
dependabot[bot] fe7b0ce535 chore(deps): bump axios from 1.18.1 to 1.20.0 (#4008)
Bumps [axios](https://github.com/axios/axios) from 1.18.1 to 1.20.0.
- [Release notes](https://github.com/axios/axios/releases)
- [Changelog](https://github.com/axios/axios/blob/v1.x/CHANGELOG.md)
- [Commits](https://github.com/axios/axios/compare/v1.18.1...v1.20.0)

---
updated-dependencies:
- dependency-name: axios
  dependency-version: 1.20.0
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-01 17:49:54 -07:00
dependabot[bot] e85c6917f7 chore(deps): bump fastify from 5.12.3 to 5.12.5 (#4009)
Bumps [fastify](https://github.com/fastify/fastify) from 5.12.3 to 5.12.5.
- [Release notes](https://github.com/fastify/fastify/releases)
- [Commits](https://github.com/fastify/fastify/commits/v5.12.5)

---
updated-dependencies:
- dependency-name: fastify
  dependency-version: 5.12.5
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-01 17:49:42 -07:00
Juan Castro 989399a857 Merge branch 'main' into juancastro/put-1896-file-and-sandboxed-iframe-origins-get-a-hard-400-and-cant 2026-10-01 20:30:41 -04:00
Juan Castro 143948ed2c Merge branch 'main' into juancastro/put-1896-file-and-sandboxed-iframe-origins-get-a-hard-400-and-cant 2026-10-01 20:30:00 -04:00
Juan Fernando Castro e48dc0cbf3 Merge pull request #4011 from HeyPuter/juancastro/put-1997-bound-grant-and-token-route-inputs
fix: bound the inputs on the grant and token routes (PUT-1997)
2026-10-01 20:29:42 -04:00
Juan Castro a67cdac380 fix: bound the inputs on the grant and token routes
Three permission routes sized their work by the request body rather than by
the route, so one call could cost far more than its rate-limit slot implies.

- `extra` and `meta` were only type-checked. They ride every row a grant
  writes (up to 16) and are re-serialised into the per-(user, app) cache on
  each miss, so they are now capped at 4 KiB.
- `/auth/create-access-token` took a `permissions` array of any length: one
  permission check and one sequential INSERT each. It now uses the same
  16-per-request cap the grant routes already had, and runs every entry
  through the validator those routes use.
- Withdrawing an app's cross-app data grants looks them up by permission
  text, and no index on `user_to_app_permissions` led with `permission`.
  Indexed per engine, mirroring what mysql_mig_22 / postgres_mig_11 /
  sqlite 0067 did for `user_to_user_permissions`.

`grant-dev-app` validated none of its input — not even the type of `extra` —
so it now runs the same validator as its user-app sibling.

Two paths could also carry a permission wider than the `varchar(255)` column
it lands in. `grantUserAppPermission` already rejected that after rewriting;
`grantDevAppPermission` now does the same, and `createAccessToken` checks it
before the session row a failing INSERT would otherwise orphan.

Caps are published in rate-limits-and-quotas.md. Every in-tree caller sends
one permission and a small `extra`, so none of them change behaviour.
2026-10-01 16:51:39 -04:00
dependabot[bot] c6da95beb0 chore(deps-dev): bump undici from 7.29.0 to 7.30.0 (#3991)
Bumps [undici](https://github.com/nodejs/undici) from 7.29.0 to 7.30.0.
- [Release notes](https://github.com/nodejs/undici/releases)
- [Commits](https://github.com/nodejs/undici/compare/v7.29.0...v7.30.0)

---
updated-dependencies:
- dependency-name: undici
  dependency-version: 7.30.0
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-01 09:52:01 -07:00
Daniel Salazar ed7b8f828a fix: kv path errors and ttl hardening, events subscription end and race fixes (#4004) 2026-10-01 09:50:34 -07:00
Juan Castro 29c6f78b7c fix: refuse opaque origins cleanly instead of a logged 400
A page the browser gives no origin to — one opened straight from disk, or an
iframe sandboxed without `allow-same-origin` — serialises its origin as
`"null"`. `appUidFromOrigin` logged that at error level on every attempt and
the GUI sent it on three endpoints, so each refusal cost a 400 and an
error-level line while the user got nothing useful.

An opaque origin cannot name an app to mint a token for, and is not a valid
`postMessage` target to deliver one to, so these clients are refused rather
than supported. Make the refusal quiet, early and legible instead:

- AuthService: drop the `console.error` and say what the caller should do.
  The accept/reject set is unchanged; a 400 is already counted per route.
- puter.js: `signIn()` rejects with `unsupported_origin` before opening
  anything, which covers implicit auth too since it routes through `signIn`.
  The load-time warning now covers sandboxed iframes, console-only — a modal
  belongs in nobody else's embed.
- GUI: a popup whose opener has no attested origin says so and closes,
  instead of rendering a blank window and wedging on a failed exchange.

Also fixes two faults this made reachable: `PuterDialog.open()` prefers the
popup branch, which reaches the `puter` global before the constructor that
assigns it has returned (`puter is not defined`, no SDK at all); and
`showModal()` throws where modals are blocked. `openNotice()` does neither.

Ordinary http(s) sign-in is unaffected.
2026-10-01 11:09:04 -04:00
Juan Fernando Castro c542e4675b Merge pull request #4000 from HeyPuter/juancastro/put-2036-shared-file-issue
fix: let a share recipient create entries in a folder shared with them
2026-10-01 09:56:09 -04:00
Daniel SalazarandReynaldi Chernando d3b46a6b86 docs: easy kv recipes (PUT-1962) (#3999)
* docs: easy kv recipes (PUT-1962)

* docs: publish the append-to-list kv recipe (PUT-1962)

* docs: bare-object add rejects with invalid_path (PUT-1962)

* refine

---------

Co-authored-by: Reynaldi Chernando <[email protected]>
2026-10-01 18:03:07 +07:00
Juan Fernando Castro fd8b8288d3 Merge pull request #3988 from HeyPuter/juancastro/put-1953-get-user-app-token-and-login-500-on-fk-failures-when-a 2026-09-30 20:28:02 -04:00
Daniel Salazar fc62ea2ee1 Revert "variable color for text in input fields (#3996)" (#4001)
This reverts commit 8862420594.
2026-09-30 16:12:19 -07:00
Juan Castro d1b11c793c Merge branch 'main' into juancastro/put-1953-get-user-app-token-and-login-500-on-fk-failures-when-a
One conflict, in workerInvoker.integration.test.ts: #3997 fixed the same
gap-marker flake this branch did, with a different barrier. Took theirs
— waiting on the `ev:qf` counter, which `#handlerFailed` writes strictly
after `discard`, so the marker is in and the lease cleared by the time it
reads 1. Their ordering also asserts the counter before flipping `answer`
back to 200, which closes the redelivery-clears-failures hole this
branch's version had removed the line to avoid.

Converted the two `vi.waitFor` calls #3997 added to `waitUntil`, the
helper this branch introduced for the rest of the file. Neither is wrong
today, since the new block fakes no timers, but a `jump()` added near
them later would hit exactly the drained-backoff trap the helper exists
for.
2026-09-30 17:49:36 -04:00
Juan Castro d6b7c8ff52 docs: drop the replica-lag testing runbook from the open-source tree
It describes our local two-server MySQL replication setup, which is
internal tooling rather than anything a self-hoster or contributor
needs. It now lives in the heyputer repo alongside the other internal
docs.

The suite it documented stays: the header comment now states the
env-var gate directly instead of pointing at a file this repo no
longer carries.
2026-09-30 17:42:04 -04:00
Daniel Salazar 4448103ef0 fix: misc events and kv issues found during recipes (#3997)
* fix: misc events and kv issues found during recipes

* test: cover handler depth on narrowed actors and created notifications

* test: fix stale legacy batch directory upload test and seat user-agent flake

* test: wait for the refusal to be recorded before claiming its gap marker
2026-09-30 14:40:47 -07:00
Juan Castro 79e496e79d test: wait on the gap marker rather than a depth that never moves
The refused-delivery test waited for `depth(subId)` to reach 1 before
claiming the marker, but `discard` removes the entry and appends the
marker in its place, so the depth is 1 before, during and after. The
wait returned at once, and the claim came back `inflight` — the refused
delivery still held the lease it was invoked on — leaving the assertion
reading `undefined`. It only passed on the slack in `invoked`'s poll
interval, which a loaded runner takes away.

Waiting on the claim itself is the barrier the depth was standing in
for. Dropping `answer = 200` with it: `beforeEach` already resets it,
and a redelivery settling before the last assertion would clear the
failure count it reads.
2026-09-30 16:36:57 -04:00
Juan Castro 2e0bf26cac Merge branch 'main' into juancastro/put-1953-get-user-app-token-and-login-500-on-fk-failures-when-a 2026-09-30 16:31:42 -04:00
Juan Castro b126cf9c5a fix: carry the owner's path into WebDAV events and expand tilde before touch
WebDAV emits the same GUI events the FS controllers do, and had the same
confusion: the payload was masked for the actor and addressed to the owner. A
recipient mounting a shared folder in Finder reproduced the disappearing-item
symptom exactly.

/touch prefixed `/` before expanding, so `~/AppData/<app>/f.txt` became a literal
`/~/...` that no longer looked like a tilde path — the shape puter.js sends for
every app-relative write answered 404.
2026-09-30 16:21:39 -04:00
Juan Castro ec4d3b31cf fix: let a share recipient create entries in a folder shared with them
A recipient addresses the owner's entries by a masked `/<owner>/<uuid>/<name>`
path. Routes that resolve an existing row unmask it on the way in, but the ones
that create a new entry had nothing to resolve and ran the ACL check against the
mask itself, which names no row — so mkdir, touch and the batch write/mkdir/
shortcut ops answered 404 `subject_does_not_exist` inside any shared folder.
They now expand client paths the same way every other legacy route does.

The GUI events carried the same confusion the other way: the payload was masked
for whoever made the request but addressed to the entry's owner, who cannot
resolve another user's mask. The owner's client dropped the row by uid and then
failed to re-add it under a path it had never heard of, so anything a recipient
touched vanished from the owner's open window until a refresh. Those events now
publish the owner's real path; responses to the actor stay masked.

Also fixes the desktop's `item.moved` handler reading `metadata` as an object
when the wire carries a JSON string: the name of a trashed item came out empty
(trashing renames the entry to its uid), and re-encoding the string left the
restore path parsing a string instead of an object. The parse the dashboard and
the explorer already open-coded is now one helper.
2026-09-30 15:28:02 -04:00
hairyEagle 8862420594 variable color for text in input fields (#3996) 2026-09-30 11:41:32 -07:00
Neal ShahandReynaldi Chernando 3b4f3f6d06 PUT-1959 PUT-1960 subdomains & worker recipes (#3993)
* PUT-1960 worker recipes

* subdomain recipes

* fixes

---------

Co-authored-by: Reynaldi Chernando <[email protected]>
2026-10-01 01:33:29 +07:00
Juan Fernando CastroandReynaldi Chernando 3d1357455f docs: teams recipe and playground examples (PUT-1966) (#3967)
* docs: add the easy teams recipe and playground examples

Covers the read-only surface an app actually has: list(), listMembers()
and listDirectory(). Adds the teams recipe tag the build validates
against, and a Teams section to the playground.

Each example handles the two states that are easy to confuse: teams
turned off on the deployment (not_found) and a team that has not opened
its directory to apps (team_not_found).

* docs: open the teams recipe with what it lets an app build

* refine

---------

Co-authored-by: Reynaldi Chernando <[email protected]>
2026-10-01 00:11:42 +07:00
Juan Castro f8d8003fea test: stop the delivery-backoff test reading holds against a moving clock
`holds one it could not answer, for longer each time` freezes `Date` so a
hold written as `now + backoff` reads back as exactly `backoff`. But every
wait in the file went through `vi.waitFor`, which advances the faked clock
once per poll — measured at 50ms a poll. Each poll between the hold being
written and read therefore ate into the value: CI read a 2s hold as 525ms
and failed, on main one run and on this branch the next.

Polling now goes through `waitUntil`, which retries on `performance.now`
and real `setTimeout` — neither faked — so the clock only moves where
`jump` moves it. The holds then read exactly 2s/4s/8s/16s, so the
assertion is an equality rather than a one-second window that a slow
enough runner could always slide out of.
2026-09-30 12:25:40 -04:00
Juan Castro 23c8f73bcb test: verify the replica-lag fix against real MySQL replication
The unit tests inject the driver error because sqlite has neither a read
replica nor foreign-key enforcement, so neither half of this bug can
occur locally — they prove the handling, not that the situation arises.

This suite runs two MySQL servers in real replication and freezes the
follower with STOP REPLICA, so the row really is gone on the primary and
really is still served by the replica, and MySQL raises the foreign-key
errors itself. Against the pre-fix code all four cases fail, including
the unhandled "Cannot add or update a child row" the reported 500s were.

Opt-in behind PUTER_TEST_REPLICA_LAG, matching the env-gated provider
integration tests and their skipUnlessEnv helper, so CI and anyone
without the containers skip it. It builds and drops its own database
rather than touching local dev data.
2026-09-30 10:02:36 -04:00
Daniel Salazar b9efac2363 fix: misc fixes from alert logs (#3995) 2026-09-30 00:51:27 -07:00
Daniel Salazar ad12da2942 feat: add gpt 6.1 sol to our direct apis (#3994) 2026-09-29 17:38:40 -07:00
Neal Shah 9aa3989e9d fix PUT-1888 (#3992) 2026-09-29 17:06:28 -04:00
dependabot[bot] cd37cf147f chore(deps): bump socket.io-client from 4.8.3 to 4.8.4 (#3982)
Bumps [socket.io-client](https://github.com/socketio/socket.io) from 4.8.3 to 4.8.4.
- [Release notes](https://github.com/socketio/socket.io/releases)
- [Changelog](https://github.com/socketio/socket.io/blob/main/CHANGELOG.md)
- [Commits](https://github.com/socketio/socket.io/compare/[email protected]@4.8.4)

---
updated-dependencies:
- dependency-name: socket.io-client
  dependency-version: 4.8.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-09-29 12:56:15 -07:00
Filip Kujundžić f60d94ef3f Merge pull request #3987 from HeyPuter/fix/put-1874-chat-test-mode
Fix chat test mode routing to fake provider
2026-09-29 19:19:44 +02:00
Juan Castro 8bce62e880 Merge branch 'main' into juancastro/put-1953-get-user-app-token-and-login-500-on-fk-failures-when-a 2026-09-29 13:13:01 -04:00
Juan Castro abd27c0192 fix: replicate tombstones, and attribute foreign-key failures correctly
Review of the first pass turned up two holes that made it ineffective.

CacheReplicationService always deleted the keys a peer broadcast and
never adopted a payload, so tombstones stopped at the region boundary:
peers dropped the marker, kept caching deleted rows off their own
replicas, and the multi-region case the fix was for stayed broken.
Tombstones are now the one payload a peer adopts — they carry no row,
only the fact of a delete.

user_to_app_permissions keys both the app and the user, and the drivers
don't all name the constraint, so a deleted *account* was being read as a
deleted app: a live app got tombstoned, the request 404'd for the wrong
reason, and the dead account was never retired, so the client retried and
re-tombstoned indefinitely. The insert site now asks the primary which
parent actually went away and answers 404 or 401 to match.

Also: re-check the tombstone after a cache write lands, since a row
cached under a live tombstone is never looked at again; extend the guard
to #refreshCache, which update() could otherwise use to undo a delete;
stop markDeleted tombstoning an address the deleted row never owned,
which was blocking the account that did own it; let the primary rather
than the tombstone decide in batched lookups; tombstone the id key when
no cached row names the others; and correct the leaf-store comment in
stores/index.ts now that SessionStore depends on UserStore.
2026-09-29 13:12:31 -04:00
dependabot[bot] 6a31b2af9e chore(deps): bump libphonenumber-js from 1.13.13 to 1.13.14 (#3984)
Bumps [libphonenumber-js](https://gitlab.com/catamphetamine/libphonenumber-js) from 1.13.13 to 1.13.14.
- [Changelog](https://gitlab.com/catamphetamine/libphonenumber-js/blob/master/CHANGELOG.md)
- [Commits](https://gitlab.com/catamphetamine/libphonenumber-js/compare/v1.13.13...v1.13.14)

---
updated-dependencies:
- dependency-name: libphonenumber-js
  dependency-version: 1.13.14
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-09-29 10:11:03 -07:00
dependabot[bot] 3ebb753eb1 chore(deps): bump socket.io from 4.8.3 to 4.8.4 (#3985)
Bumps [socket.io](https://github.com/socketio/socket.io) from 4.8.3 to 4.8.4.
- [Release notes](https://github.com/socketio/socket.io/releases)
- [Changelog](https://github.com/socketio/socket.io/blob/main/CHANGELOG.md)
- [Commits](https://github.com/socketio/socket.io/compare/[email protected]@4.8.4)

---
updated-dependencies:
- dependency-name: socket.io
  dependency-version: 4.8.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-09-29 10:08:32 -07:00
Juan Fernando CastroandReynaldi Chernando ee4d55027d docs: easy perms recipe (PUT-1965) (#3972)
* docs: add the easy perms recipe

Covers the supported surface: request() and check(). Adds the perms
recipe tag the build validates against.

* docs: add a playground example per operation in the perms recipe

One runnable example for each section: request a folder, request the
email address, request apps or subdomains, and check before prompting.

* docs: open the perms recipe with what it lets an app build

* docs: ground the perms examples in the UI that would ask

A photo editor exporting to Pictures, a checkout emailing a receipt, a
launcher listing apps, and a settings toggle that reads its own state
without raising a prompt.

* refine

---------

Co-authored-by: Reynaldi Chernando <[email protected]>
2026-09-30 00:00:34 +07:00
dependabot[bot] 7eb9ace733 chore(deps-dev): bump prettier from 3.9.8 to 3.9.9 (#3986)
Bumps [prettier](https://github.com/prettier/prettier) from 3.9.8 to 3.9.9.
- [Release notes](https://github.com/prettier/prettier/releases)
- [Changelog](https://github.com/prettier/prettier/blob/main/CHANGELOG.md)
- [Commits](https://github.com/prettier/prettier/compare/3.9.8...3.9.9)

---
updated-dependencies:
- dependency-name: prettier
  dependency-version: 3.9.9
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-09-29 09:51:06 -07:00
dependabot[bot] 14ae6a69ee chore(deps): bump postal-mime from 3.0.0 to 3.0.1 (#3983)
Bumps [postal-mime](https://github.com/postalsys/postal-mime) from 3.0.0 to 3.0.1.
- [Release notes](https://github.com/postalsys/postal-mime/releases)
- [Changelog](https://github.com/postalsys/postal-mime/blob/master/CHANGELOG.md)
- [Commits](https://github.com/postalsys/postal-mime/compare/v3.0.0...v3.0.1)

---
updated-dependencies:
- dependency-name: postal-mime
  dependency-version: 3.0.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-09-29 09:50:45 -07:00
Juan Fernando CastroandReynaldi Chernando 3b00cd2d14 docs: easy fs sharing recipe (PUT-1958) (#3973)
* docs: add the easy fs sharing recipe

share(), listShared(), getShares() and unshare() — the fs ground the
existing recipes do not already cover.

* docs: add a playground example per operation in the fs sharing recipe

One runnable example for each section of the recipe: share, mode,
listShared, getShares and unshare.

* docs: open the sharing recipe with what it lets an app build

* docs: ground the sharing examples in a document-editor flow

Each example now sits in the UI that would call it — a Share button, a
role picker, a shared-with-me inbox, an access list, ending a review.

* remove redundant examples and refine recipe

---------

Co-authored-by: Reynaldi Chernando <[email protected]>
2026-09-29 23:47:23 +07:00
Juan Castro 785304e807 fix: stop deleted apps and users from being resurrected by a lagging replica
A delete lands on the primary and invalidates the cache, but a concurrent
request that misses the cache reads a replica that hasn't applied it yet,
finds the row, and writes it back under a fresh TTL — 24h for apps, 15m
for users. Every write referencing that resurrected row then fails its
foreign key and escapes as a raw driver error, so the route 500s until
the TTL expires. Origin-bootstrapped app uids are deterministic, so the
same origin keeps hitting the stale row and never re-bootstraps.

Deletes now tombstone their cache keys before dropping them. While a
tombstone is live the read path goes to the primary and the write path
refuses to cache; re-creating a row clears it. As a backstop, a write
rejected for a missing parent is answered as one: 404 for a vanished app,
401 for a vanished account, with the stale entry retired either way.

Also fixes cascadeDelete's identity lookup, which selected a column some
schemas lack and so threw into its catch, skipping the invalidation
entirely; and drops SessionStore's private copy of isUniqueViolation in
favour of the shared helper.
2026-09-29 10:16:48 -04:00
Juan Fernando Castro 8e31d9149a Merge pull request #3966 from HeyPuter/juancastro/put-1994-group-scan-cycle
fix: stop group permission scans re-entering the sharer's own grant (PUT-1994)
2026-09-29 10:12:54 -04:00
Juan Castro 9539c7b553 test: cover team-share access through the real authorization path
The existing team-share suite resolves shares through listings and store
reads, neither of which enters the permission scan, so a scan that never
terminates leaves it green. These ask the question the product asks —
can this member open the file — via ACLService.check.
2026-09-29 09:41:05 -04:00
Juan Fernando Castro 27c4873042 Merge pull request #3971 from HeyPuter/juancastro/put-2033-team-create-read-after-write
fix: read a team back from the primary after writing it (PUT-2033)
2026-09-29 09:06:43 -04:00
404oops 0b1a4283f5 fix(ai): honor chat test mode 2026-09-29 12:17:33 +02:00
Daniel Salazar 1cc091206c fix: PUT-1885 PUT 1950 (#3981)
* fix: app uuid fixes

* fix: bound kv update/add createPaths writes (PUT-1885)

* fix: upload progress socket events on the v2 tracker (PUT-1950)
2026-09-28 23:38:34 -07:00
Daniel Salazar 09887747e0 fix: app uuid fixes (#3980) 2026-09-28 19:43:23 -07:00
Daniel Salazar f80d4ae136 fix: events docs (#3968)
* fix: events docs

* fix: docs
2026-09-28 17:24:37 -07:00
Filip KujundžićandClaude Opus 5.5 42e0c28d0a feat(ai): add Claude Sonnet 5.5 (#3970)
Add claude-sonnet-5-5 ($2/$10 per MTok, 1M context, 128K output) and move
the bare claude-sonnet / claude-sonnet-latest aliases to it. Sonnet 5.5
rejects non-default sampling params and disabled thinking, so it joins the
temperature-omit, effort, and adaptive-thinking gates.

Co-authored-by: Claude Opus 5.5 <[email protected]>
2026-09-28 15:29:21 -07:00
Juan Castro d888377138 Merge remote-tracking branch 'origin/main' into juancastro/put-2033-team-create-read-after-write 2026-09-28 17:20:43 -04:00
Juan Castro 8a53d14540 fix: read a team back from the primary after writing it (PUT-2033)
create() inserted on the primary then read the row back through the
replica, so a replica that had not caught up answered the same as a team
that was not there and the call threw. Nothing rolled the insert back,
and the owner membership was never written, leaving a team its owner
could not see but which still counted against their cap.

Read back through pread in create() and update(), seed the row cache
with the result, and delete the row if the read-back still finds
nothing. #cached no longer stores a null, which would otherwise hold a
lagging replica's answer for the full TTL.
2026-09-28 16:55:19 -04:00
Juan Fernando Castro 12e6bc3d71 Merge pull request #3950 from HeyPuter/juancastro/put-1988-team-force-2fa
feat: let a team require two-factor authentication of its accounts (PUT-1988)
2026-09-28 16:52:34 -04:00
Filip Kujundžić 359d68e51f Merge pull request #3949 from HeyPuter/fix/ai-ocr-model-sync-and-normalization 2026-09-28 21:23:28 +02:00
Filip Kujundžić 020e61065f Merge pull request #3947 from AnujShrivastava01/anuj/fix/remove-deprecated-ai-chat-models 2026-09-28 21:22:33 +02:00
Juan Castro 480d9d032b Merge remote-tracking branch 'origin/main' into juancastro/put-1988-team-force-2fa
# Conflicts:
#	src/backend/clients/database/SqliteDatabaseClient.test.ts
#	src/backend/clients/database/SqliteDatabaseClient.ts
#	src/backend/clients/database/migrations/mysql/mysql_mig_42.sql
#	src/backend/clients/database/migrations/postgres/postgres_mig_31.sql
2026-09-28 10:24:39 -04:00
Juan Castro 528eb192b5 fix: stop group permission scans re-entering the sharer's own grant
The group path recursed without the anti-cycle state, so a grant issued
into a group the issuer belongs to resolved back into itself and the
scan did not terminate. Thread the scan state through the group scanner
and skip an issuer already on the path, matching the user-to-user path.

Readings produced under a non-empty path are pruned for that path only,
so they are no longer written to or served from the shared scan cache.
2026-09-28 10:05:54 -04:00
Rupanjana ChoudhuryandDaniel Salazar ca52b5236c Fix/3638 shared link account (#3845)
* fix: open shared links with the intended account

* refactor: validate shared account hints in the GUI

* fix: scope the shared-link account hint to plain top-level visits

The hint no longer switches accounts in popups, embeds or action flows, and
only counts next to a share path the GUI would open. It is dropped from the
URL before any switch or prompt, so a reload never asks again. An unsaved
recipient gets a dismissible choice (sign in, or continue as the current
account) instead of a picker with no way out.

---------

Co-authored-by: Daniel Salazar <[email protected]>
2026-09-27 18:25:28 -07:00
Neal Shah 153f581011 fix PUT-1872 (#3952) 2026-09-25 22:34:53 -04:00
Neal Shah 1fc2eae1ae self hosted mail (#3932)
* self hosted mail

* update lock

* address daniel review
2026-09-25 21:48:15 -04:00
Daniel Salazar dcb61bc4fd fix: PUT-1824 + PUT-1884 + PUT-1952 + PUT-1827 (#3951) 2026-09-25 17:47:03 -07:00
Juan Fernando Castro b445680f10 Merge pull request #3937 from HeyPuter/juancastro/put-1879-oidc-popup-return-proof-binding
fix(auth): bind the OIDC popup-return proof to its purpose, browser, and popup
2026-09-25 18:04:59 -04:00
Juan Castro 95ee796157 Merge branch 'main' into juancastro/put-1879-oidc-popup-return-proof-binding 2026-09-25 17:22:12 -04:00
Nariman JelvehandDaniel Salazar 3d3678a555 feat: let the Contact Us form carry screenshots and recordings (#3563)
* feat: let the Contact Us form carry screenshots and recordings

The form doubles as our bug-report channel, and the bugs worth reporting
are often the ones that need a picture: a visual glitch, or something that
takes five steps to reach. Up to 5 images or videos may now ride along
with the message, delivered as attachments on the support email.

Nothing the client says about a file is believed. The endpoint takes bare
base64 and re-derives all three of the things that matter from the decoded
bytes:

- Type, sniffed from magic numbers and matched against an allow-list of
  PNG/JPEG/GIF/WebP and MP4/QuickTime/WebM. A declared MIME is never read,
  so it cannot smuggle anything past the list. SVG is excluded on purpose
  (it carries script, and support tooling renders what it is sent), as are
  the non-video brands that share MP4's `ftyp` box -- HEIC, M4A, JPEG 2000.
- The file name, reduced to a display label with the extension taken from
  the sniffed type. PNG bytes named `payload.html` arrive as
  `payload.png`; a name can never carry the CR/LF that would break out of
  a Content-Disposition header, nor the bidi overrides that make
  `report<RLO>gnp.exe` render as `report.exe.mp4`.
- Size: 10 MB per file, 15 MB per submission, counted on decoded bytes.
  Encoded length is capped before decoding, so an oversized payload costs
  a length check rather than a 10 MB allocation. The total stays well
  under the 25 MB most providers enforce, since the outgoing mail base64s
  these again.

Base64 is decoded strictly, reusing the round-tripping decoder that
already guards app icons -- `Buffer.from(s, 'base64')` silently drops
characters it does not recognise, and the round-trip is what rejects
bytes smuggled after the payload. That decoder and the image sniffer move
from appIcon.ts to a new mediaSniff.ts, which gains the video counterpart.

One request is now worth megabytes of parsing and outbound mail, so the
existing per-user rate limit gains a per-IP backstop (which the per-user
counter cannot see through freshly minted accounts), a concurrency cap,
and a Content-Length gate that refuses an impossible body before anything
decodes it.

Payloads are not stored. They ride the email; the new
`feedback.attachments` column records names, types and sizes only, so an
abusive submission stays attributable once the mail has been dealt with.

Also fixes the form posting an empty message when Send was pressed with
nothing typed, and surfaces submit failures instead of leaving the button
disabled with no explanation.

* fix: stream Contact Us attachments as multipart instead of base64 JSON

Base64 in a JSON body went through the global JSON parser before the route
ran: raw buffer, rawBody copy, decoded string, parsed strings, then decoded
Buffers plus a re-encode for the strict check. A max-size submission peaked
around 110-135 MB of heap for 15 MB of files, and the route's Content-Length
check ran after all of it.

Attachments now arrive as multipart/form-data, which the global parser skips.
Auth, rate limit, concurrency and the Content-Length 413 all run before the
body is read, and busboy enforces the count, per-file and total caps while
streaming, so only the decoded file bytes are held (~16 MB peak). On a broken
limit the reader stops and the response closes the connection.

JSON stays supported for message-only posts; a JSON `attachments` field is
rejected.

* fix: deliver Contact Us 413s instead of resetting the upload

Closing the socket on a mid-stream limit sent the 413 and then reset the
connection with the client still uploading, so the client could see a
reset instead of the error. After a limit trips, the rest of the body is
now read and discarded, bounded by the body budget; past the budget the
request is destroyed.

Adds an HTTP-level test through the full middleware stack (FormData
upload, unauthenticated refusal, 413 on declared length before the body
is sent, 413 on a chunked upload over the per-file cap), trims comments
to the AGENTS.md length rule, and drops a box-drawing test divider.

---------

Co-authored-by: Daniel Salazar <[email protected]>
2026-09-25 14:19:35 -07:00
hairyEagle a9ee3d1ea6 fix(selfhost): configure bucket CORS in s3-init so browser uploads work (#3934)
Browser uploads (Dev Center deploy, puter.fs.upload) PUT file bytes
directly to presigned URLs on the public S3 endpoint. That cross-origin
PUT is preflighted, and the bucket had no CORS rules, so RustFS answered
without Access-Control-Allow-Origin and the browser blocked the upload.

s3-init now runs put-bucket-cors on every boot after ensuring the bucket
exists (idempotent). Origins are open because the presigned URL is the
credential and the SDK sends no cookies. Document how to re-run and
verify in doc/self-hosting.md.
2026-09-25 12:12:02 -07:00
Juan Castro ee3f928b0e fix: annotate the seat-lookup catch so the baselined typecheck passes 2026-09-25 14:39:23 -04:00
Juan Castro 176c8848f5 fix: let a seat enrol in 2FA from the console, and hold it once required
The Security tab only rendered the 2FA card for accounts with a confirmed
email, so a seat — provisioned without one by design — had no way to
enrol from the UI at all. It was the GUI twin of the enable-route check
fixed earlier: the rule could be required of accounts that could not
satisfy it. The card now shows for a seat too, giving it the ordinary
authenticator flow: QR, manual secret, six-digit code, recovery codes.

Disabling is refused while the seat's team requires 2FA. Without that a
member could switch off the factor their team mandates and lock
themselves out of everything until they enrolled again.
2026-09-25 14:31:58 -04:00
Juan Castro beaf4a9a7c docs: record the team 2FA rule where the other member-facing gates are
No method pages: requiring 2FA and resetting a member's factor are team
administration, which the docs deliberately do not publish — only
`list`, `listMembers` and `listDirectory` are app-callable.

But an app acting for a gated seat does meet `two_factor_required`, and
an undocumented 403 leaves it nothing to go on. It goes beside
`password_change_required` in rate-limits-and-quotas, which already
explains what a team's administration does to a member's requests.
2026-09-25 14:18:30 -04:00
404oopsandClaude Opus 5.5 d36da7522b fix(ai-ocr): address review: playground examples, Textract options, inline size
Adds the Mistral and document-annotation img2txt examples to the
playground, states that Textract takes no options, and drops the
normalize mentions from the img2txt docs and types.

The SDK now checks Mistral inline sources against the size a base64
upload can reach inside the 50 MB JSON body (36 MB) instead of the
10 MB Textract limit; 50 MB let 38-50 MB inputs through to a body-parser
413. Retired-model messages no longer cite vendor dates.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
2026-09-25 18:56:29 +02:00
Daniel Salazar e6cfe8c492 fix: codeowners (#3948) 2026-09-25 09:23:26 -07:00
404oopsandClaude Opus 5.5 eef8258539 fix(ai-ocr): sync OCR models and prices, route by model, cap input size
Model sync against Mistral's listing, docs and a live call per id:
mistral-ocr-latest now serves OCR 4.1, so the catalog pins it (with
mistral-ocr-4 and the retired-but-redirected 2503) to mistral-ocr-4-1,
adds mistral-ocr-4-0 and OCR 3 (mistral-ocr-2512 / -3 / -3-0), and
rejects the retired mistral-ocr-2505 before any upstream call. Mistral
usage is metered per model at vendor rates: $4/1000 pages for OCR 4.x
with $5 annotations, $2 and $3 for OCR 3, replacing the flat $1 and $3.

Textract now calls DetectDocumentText instead of AnalyzeDocument with
LAYOUT. The layout feature cost $4/1000 pages while we billed the
$1.50 detect rate, and it only added text-less LAYOUT_* blocks that put
blank lines into img2txt output. Multi-page PDFs and unsupported
formats become a 400 that names Mistral as the alternative.

Request shape: a model name alone now selects its provider, so calls
differ only by model; an explicit provider still picks its default and
must agree with the model. Mistral options gain tableFormat,
extractHeader, extractFooter and documentAnnotationPrompt; annotation
formats accept the REST (json_schema.schema) or SDK
(jsonSchema.schemaDefinition) spelling and are validated before
billing. Non-image documents such as DOCX go as document_url.

Response shape: both providers return model, text and blocks with a
0-based page; Mistral also returns document_annotation, which was
billed but dropped. img2txt resolves to that annotation when
documentAnnotationFormat is set and to the text otherwise; the return
is a string for every model, so normalize has no effect on it.

Input size: OCR input is capped at the provider limit (10 MB Textract,
50 MB Mistral), and web inputs in loadFileInput are streamed under the
cap instead of buffered whole, which also bounds speech2txt URLs.

Docs: img2txt models, options, input limits, return value and errors;
OCR limits in the quotas page.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
2026-09-25 18:19:54 +02:00
Juan Castro 3105378fdc fix: let a team seat enrol in 2FA, and fit the rule's note on its card
Found by driving the flow against a live instance: a provisioned seat
could reach `configure-2fa/setup` but never `enable`, which demands a
confirmed email. Seats are provisioned without one on purpose (PUT-1792,
the team being the trust anchor), so requiring 2FA of a team would have
locked out precisely the accounts the rule is for — permanently, since a
seat cannot enrol and the rule never lifts.

The check now passes for an account a team provisioned; every other
account still needs its address confirmed, with tests either way.

The card's note was also a line too long and rendered clipped, so it says
the same thing in the space the layout gives it.
2026-09-25 11:58:32 -04:00
Juan Castro 695a26b272 feat: surface the team 2FA rule in the console and SDK (PUT-1988)
An owner toggle beside the directory one, confirmed on the way on since
it locks out every account without 2FA until they enrol, and unconfirmed
on the way off since that only lifts a requirement.

A per-member Reset 2FA action, shown only while the rule is on — with it
off a member can clear their own factor and needs nobody's help. The
confirmation says what it does and what it leaves alone: signed out
everywhere, told about it, password unchanged.

`require2fa` rides the Team shape and `update()`, and `resetTwoFactor()`
joins the module beside `resetPassword()`, whose docblock already says
2FA is left alone — the two now read as the pair they are.
2026-09-25 11:28:59 -04:00
Juan Castro cab70254b1 feat: let a team owner reset a seat's second factor (PUT-1988)
Forcing 2FA makes lockout materially likelier, and nothing could clear a
factor except the account holding it — `resetMemberPassword` leaves 2FA
alone on purpose, and admin only displays it. A seat that lost both its
authenticator and its recovery codes had no way back.

Its own route and its own audit action rather than folded into the
password reset, so that method keeps the property its comment claims: a
password reset alone is not takeover. The two together are, and for an
account the team created and pays for that is the team's to do — but it
has to be chosen and recorded, not arrived at sideways.

Bounded to `org_owned = 1`: a joined member's own login is not a team's
to reach into. The member is told, and signed out everywhere.
2026-09-25 11:15:23 -04:00
Juan Castro db1a95260a feat: let a team require 2FA of the accounts it provisioned (PUT-1988)
`group.require_2fa`, toggled by the owner through `PUT /teams/:uid` and
recorded in the audit log like `directory_enabled` — it changes what
every seat must hold to sign in, so it is not something a team should be
able to alter silently.

Enforced by deriving, not by stamping a flag on the user: a stamp left
behind by a member leaving, or by the rule being turned off, would lock
someone out of their own account with no team left to clear it. The seat
row already joins `group` and is already read per request, so it carries
the flag and the gate costs nothing extra. Toggling busts those rows, or
turning the rule off would leave members locked out until the TTL lapsed.

The gate rides the same `allowUnconfirmed` opt-out as the other pending-
requirement gates, so the routes a gated member must still reach stay
reachable — `configure-2fa` among them, which would otherwise be walled
off behind the very requirement it clears.

Scoped to `org_owned = 1`: the owner is not a seat of their own team, and
a joined member is an ordinary self-paying account whose personal login
is not a team's to dictate. The owner is held to the rule at the point it
matters — they cannot turn it on without 2FA of their own.
2026-09-25 10:35:24 -04:00
Juan Castro ca48477db9 Merge branch 'main' into juancastro/put-1879-oidc-popup-return-proof-binding 2026-09-24 12:19:59 -04:00
Juan Castro d73933b8a5 fix(auth): bind the OIDC popup-return proof to its purpose, browser, and popup
Tokens under the `oidc-state` scope share one signing key, so the payload is
what says which job a token belongs to. The popup-return proof now carries a
`purpose` claim, and each verifier accepts exactly one kind of token.

The return leg also sets a single-use companion cookie, mirroring the nonce
`/start` already uses, and stamps the action its redirect lands on. Redeeming
a proof requires the matching cookie — consumed on success only, so a rejected
call can't invalidate an in-flight return — and the popup honors only a proof
minted for its own action.

The cookie is host-only, so `/auth/oidc/verify-popup-return` is also served on
the GUI origin and the popup redeems it same-origin. The return leg always
lands the popup on `config.origin`, where both the cookie and the route live.
2026-09-23 17:21:35 -04:00
1374 changed files with 195445 additions and 53444 deletions

No files matched your search

+5
View File
@@ -16,3 +16,8 @@ MARIADB_PASSWORD=replace-with-strong-password
S3_ACCESS_KEY=puter
S3_SECRET_KEY=replace-with-strong-secret
S3_BUCKET=puter-local
# ── Inbound SMTP (compose profile `smtp`) ------------------------------
# Host port mapped to the receiver's 2525. 25 is what an MX record means;
# many hosting providers block it.
SMTP_PORT=25
+57 -46
View File
@@ -1,61 +1,72 @@
# Default owner, including backend and SDK areas without a specialist below.
# The last matching rule takes precedence.
* @Salazareo
- @Salazareo
# GUI
/src/gui/ @jelveh @Salazareo
# Documentation, developer site, and CLI
/doc/ @reynaldichernando @Salazareo
/src/docs/ @reynaldichernando @Salazareo
/src/dev-center/ @reynaldichernando @Salazareo
/src/cli/ @reynaldichernando @Salazareo
/doc/ @reynaldichernando @Salazareo
/src/docs/ @reynaldichernando @Salazareo
/src/dev-center/ @reynaldichernando @Salazareo
/src/cli/ @reynaldichernando @Salazareo
# Backend AI
/src/backend/controllers/puterai/ @404oops @ProgrammerIn-wonderland @Salazareo
/src/backend/drivers/ai-\*/ @404oops @ProgrammerIn-wonderland @Salazareo
# Backend: AI, hosting, workers, networking, and WebDAV
/src/backend/controllers/puterai/ @ProgrammerIn-wonderland @Salazareo
/src/backend/controllers/hosting/ @ProgrammerIn-wonderland @Salazareo
/src/backend/controllers/webdav/ @ProgrammerIn-wonderland @Salazareo
/src/backend/controllers/wisp/ @ProgrammerIn-wonderland @Salazareo
/src/backend/drivers/ai-*/ @ProgrammerIn-wonderland @Salazareo
/src/backend/drivers/util/ @ProgrammerIn-wonderland @Salazareo
/src/backend/drivers/workers/ @ProgrammerIn-wonderland @Salazareo
/src/backend/services/localworker/ @ProgrammerIn-wonderland @Salazareo
/src/backend/stores/subdomain/ @ProgrammerIn-wonderland @Salazareo
/extensions/workerSandbox* @ProgrammerIn-wonderland @Salazareo
/src/worker/ @ProgrammerIn-wonderland @Salazareo
/src/worker-types/ @ProgrammerIn-wonderland @Salazareo
/src/backend/controllers/peer/ @ProgrammerIn-wonderland @Salazareo
/src/backend/controllers/hosting/ @ProgrammerIn-wonderland @Salazareo
/src/backend/controllers/webdav/ @ProgrammerIn-wonderland @Salazareo
/src/backend/controllers/wisp/ @ProgrammerIn-wonderland @Salazareo
/src/backend/drivers/util/ @ProgrammerIn-wonderland @Salazareo
/src/backend/drivers/workers/ @ProgrammerIn-wonderland @Salazareo
/src/backend/services/localworker/ @ProgrammerIn-wonderland @Salazareo
/src/backend/stores/subdomain/ @ProgrammerIn-wonderland @Salazareo
/extensions/workerSandbox\* @ProgrammerIn-wonderland @Salazareo
/src/worker/ @ProgrammerIn-wonderland @Salazareo
/src/worker-types/ @ProgrammerIn-wonderland @Salazareo
/src/backend/controllers/peer/ @ProgrammerIn-wonderland @Salazareo
# Backend: sharing, permissions, and teams
/src/backend/controllers/share/ @jfcastro92 @Salazareo
/src/backend/controllers/team/ @jfcastro92 @Salazareo
/src/backend/services/share/ @jfcastro92 @Salazareo
/src/backend/services/permission/ @jfcastro92 @Salazareo
/src/backend/services/acl/ @jfcastro92 @Salazareo
/src/backend/services/team/ @jfcastro92 @Salazareo
/src/backend/services/apps/AppPermissionService* @jfcastro92 @Salazareo
/src/backend/stores/share/ @jfcastro92 @Salazareo
/src/backend/stores/permission/ @jfcastro92 @Salazareo
/src/backend/stores/group/ @jfcastro92 @Salazareo
/src/backend/stores/team/ @jfcastro92 @Salazareo
/src/backend/controllers/share/ @jfcastro92 @Salazareo
/src/backend/controllers/team/ @jfcastro92 @Salazareo
/src/backend/services/share/ @jfcastro92 @Salazareo
/src/backend/services/permission/ @jfcastro92 @Salazareo
/src/backend/services/acl/ @jfcastro92 @Salazareo
/src/backend/services/team/ @jfcastro92 @Salazareo
/src/backend/services/apps/AppPermissionService\* @jfcastro92 @Salazareo
/src/backend/stores/share/ @jfcastro92 @Salazareo
/src/backend/stores/permission/ @jfcastro92 @Salazareo
/src/backend/stores/group/ @jfcastro92 @Salazareo
/src/backend/stores/team/ @jfcastro92 @Salazareo
# SDK: AI, hosting, workers, and networking
/src/puter-js/src/modules/ai/ @ProgrammerIn-wonderland @Salazareo
/src/puter-js/src/modules/hosting/ @ProgrammerIn-wonderland @Salazareo
/src/puter-js/src/modules/Workers* @ProgrammerIn-wonderland @Salazareo
/src/puter-js/src/modules/networking/ @ProgrammerIn-wonderland @Salazareo
/src/puter-js/src/lib/networkUtils* @ProgrammerIn-wonderland @Salazareo
/src/puter-js/tests/api/suites/ai.suite.ts @ProgrammerIn-wonderland @Salazareo
/src/puter-js/tests/api/suites/hosting.suite.ts @ProgrammerIn-wonderland @Salazareo
/src/puter-js/tests/api/suites/workers.suite.ts @ProgrammerIn-wonderland @Salazareo
/src/puter-js/tests/api/suites/net.suite.ts @ProgrammerIn-wonderland @Salazareo
/src/puter-js/src/modules/Peer* @ProgrammerIn-wonderland @Salazareo
/src/puter-js/src/modules/ai/ @ProgrammerIn-wonderland @Salazareo
/src/puter-js/src/modules/hosting/ @ProgrammerIn-wonderland @Salazareo
/src/puter-js/src/modules/Workers* @ProgrammerIn-wonderland @Salazareo
/src/puter-js/src/modules/networking/ @ProgrammerIn-wonderland @Salazareo
/src/puter-js/src/lib/networkUtils* @ProgrammerIn-wonderland @Salazareo
/src/puter-js/tests/api/suites/ai.suite.ts @ProgrammerIn-wonderland @Salazareo
/src/puter-js/tests/api/suites/hosting.suite.ts @ProgrammerIn-wonderland @Salazareo
/src/puter-js/tests/api/suites/workers.suite.ts @ProgrammerIn-wonderland @Salazareo
/src/puter-js/tests/api/suites/net.suite.ts @ProgrammerIn-wonderland @Salazareo
/src/puter-js/src/modules/Peer\* @ProgrammerIn-wonderland @Salazareo
# SDK: sharing and permissions
/src/puter-js/src/modules/perms/ @jfcastro92
/src/puter-js/src/modules/FileSystem/operations/share* @jfcastro92 @Salazareo
/src/puter-js/src/modules/FileSystem/operations/unshare* @jfcastro92 @Salazareo
/src/puter-js/src/modules/FileSystem/operations/getShares* @jfcastro92 @Salazareo
/src/puter-js/src/modules/FileSystem/operations/listShared* @jfcastro92 @Salazareo
/src/puter-js/tests/api/suites/perms.suite.ts @jfcastro92 @Salazareo
/src/puter-js/tests/api/suites/sharing.suite.ts @jfcastro92 @Salazareo
/src/puter-js/src/modules/perms/ @jfcastro92
/src/puter-js/src/modules/FileSystem/operations/share* @jfcastro92 @Salazareo
/src/puter-js/src/modules/FileSystem/operations/unshare* @jfcastro92 @Salazareo
/src/puter-js/src/modules/FileSystem/operations/getShares* @jfcastro92 @Salazareo
/src/puter-js/src/modules/FileSystem/operations/listShared* @jfcastro92 @Salazareo
/src/puter-js/tests/api/suites/perms.suite.ts @jfcastro92 @Salazareo
/src/puter-js/tests/api/suites/sharing.suite.ts @jfcastro92 @Salazareo
@@ -22,7 +22,7 @@ jobs:
- name: Set up Node.js
uses: actions/setup-node@v7
with:
node-version: '24'
node-version: '26'
cache: 'npm'
- name: Install dependencies
+120 -58
View File
@@ -1,17 +1,21 @@
name: Backend Tests
# Always triggers so it can be a required PR check; the `changes` job below
# Always triggers on PRs so it can be a required check; the `changes` job below
# decides whether backend/extension paths actually changed (test-only changes
# match the same globs, since tests are co-located with the code), and every
# other job is skipped — not absent — when they didn't, which still satisfies
# a required check. puter.js SDK changes are covered by puterjs-tests.yaml.
# match the same globs, since tests are co-located with the code), and `test`
# still reports success when they didn't. Pushes to main produce the coverage
# PRs compare against. puter.js SDK changes are covered by puterjs-tests.yaml.
on:
pull_request:
types: [opened, synchronize, reopened]
push:
branches: [main]
permissions:
contents: read
pull-requests: write
# Reads main's coverage artifact for the PR comparison.
actions: read
jobs:
changes:
@@ -42,12 +46,9 @@ jobs:
# The consuming repo runs an equivalent gate, but only once someone bumps the
# submodule pointer — too late to keep the error off this repo's default
# branch. Hence a gate here, on this repo's own pull requests.
#
# Its own job rather than a step in `test`: that one runs a base/PR matrix for
# coverage comparison, and the check only needs the PR ref, once.
typecheck:
needs: changes
if: needs.changes.outputs.backend == 'true'
if: github.event_name == 'pull_request' && needs.changes.outputs.backend == 'true'
runs-on: ubuntu-latest
steps:
- name: Checkout repository
@@ -56,7 +57,7 @@ jobs:
- name: Set up Node.js
uses: actions/setup-node@v7
with:
node-version: '24'
node-version: '26'
cache: 'npm'
- name: Install dependencies
@@ -65,85 +66,146 @@ jobs:
- name: Type check (new errors only)
run: npm run typecheck
# `needs.changes` gates the steps below, not this job, on purpose: a
# job-level `if` on a matrix job skips the whole job as one unexpanded
# check ("test (${{ matrix.artifact }})", literally, since there's no
# matrix context to fill it in) instead of the two real per-artifact
# checks a required rule pins to ("test (base)" / "test (pr)") — so the
# required check never gets reported and the PR blocks forever. Letting
# the matrix always expand keeps the check names stable either way.
test:
# The suite is CPU-bound, so it's split across runners. Each shard writes a
# blob report (results plus coverage) that the `coverage` job merges.
test-shard:
needs: changes
# Overrides the default matrix naming, which would otherwise bake the
# base/head branch name (matrix.ref) into the check name — making it a
# different string on every PR and impossible to pin as a required check.
name: test (${{ matrix.artifact }})
if: needs.changes.outputs.backend == 'true'
name: test shard ${{ matrix.shard }}/${{ strategy.job-total }}
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
include:
- ref: ${{ github.base_ref }}
artifact: base
- ref: ${{ github.head_ref }}
artifact: pr
shard: [1, 2, 3, 4]
steps:
- name: Checkout ${{ matrix.ref }}
if: needs.changes.outputs.backend == 'true'
- name: Checkout
uses: actions/checkout@v7
with:
ref: ${{ matrix.ref }}
repository: ${{ github.event.pull_request.head.repo.full_name }}
ref: ${{ github.event.pull_request.head.sha || github.sha }}
repository: ${{ github.event.pull_request.head.repo.full_name || github.repository }}
- name: Set up Node.js
if: needs.changes.outputs.backend == 'true'
uses: actions/setup-node@v7
with:
node-version: '24'
node-version: '26'
cache: 'npm'
- name: Install dependencies
if: needs.changes.outputs.backend == 'true'
run: npm ci
- name: Run backend tests with coverage
if: needs.changes.outputs.backend == 'true'
run: npm run test:backend -- --coverage
run: >-
npm run test:backend --
--coverage
--shard=${{ matrix.shard }}/${{ strategy.job-total }}
--reporter=default
--reporter=blob
env:
CI: 'true'
- name: Upload blob report
if: ${{ !cancelled() }}
uses: actions/upload-artifact@v7
with:
name: backend-blob-${{ matrix.shard }}
path: .vitest-reports/
include-hidden-files: true
retention-days: 1
# Required check, so its PR name is pinned. A skipped job would satisfy it,
# so this always runs and fails on any shard that didn't pass.
test:
needs: [changes, test-shard]
if: ${{ !cancelled() }}
name: test (${{ github.event_name == 'push' && 'main' || 'pr' }})
runs-on: ubuntu-latest
steps:
- name: Check shard results
env:
CHANGES_RESULT: ${{ needs.changes.result }}
BACKEND_CHANGED: ${{ needs.changes.outputs.backend }}
SHARDS_RESULT: ${{ needs.test-shard.result }}
run: |
if [ "$CHANGES_RESULT" != "success" ]; then
echo "changes job: $CHANGES_RESULT"
exit 1
fi
if [ "$BACKEND_CHANGED" != "true" ]; then
echo "No backend changes."
exit 0
fi
if [ "$SHARDS_RESULT" != "success" ]; then
echo "test shards: $SHARDS_RESULT"
exit 1
fi
coverage:
needs: [changes, test-shard]
if: ${{ !cancelled() && needs.test-shard.result == 'success' }}
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v7
with:
ref: ${{ github.event.pull_request.head.sha || github.sha }}
repository: ${{ github.event.pull_request.head.repo.full_name || github.repository }}
- name: Set up Node.js
uses: actions/setup-node@v7
with:
node-version: '26'
cache: 'npm'
- name: Install dependencies
run: npm ci
- name: Download blob reports
uses: actions/download-artifact@v8
with:
pattern: backend-blob-*
path: .vitest-reports
merge-multiple: true
- name: Merge shard coverage
run: npx vitest run --config src/backend/vitest.config.ts --merge-reports --coverage
env:
CI: 'true'
- name: Upload coverage artifact
if: needs.changes.outputs.backend == 'true' && !cancelled()
uses: actions/upload-artifact@v7
with:
name: backend-coverage-${{ matrix.artifact }}
name: backend-coverage-${{ github.event_name == 'push' && 'main' || 'pr' }}
path: src/backend/coverage/
retention-days: 14
retention-days: ${{ github.event_name == 'push' && 30 || 14 }}
report-coverage:
needs: [changes, test]
if: always() && needs.changes.outputs.backend == 'true'
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v7
- name: Download PR coverage
uses: actions/download-artifact@v8
with:
name: backend-coverage-pr
path: src/backend/coverage
- name: Download base coverage
uses: actions/download-artifact@v8
with:
name: backend-coverage-base
path: coverage-base
# Latest main coverage from this repo's own runs; a fork can't publish
# under this repository id.
- name: Download main coverage
id: base
if: github.event_name == 'pull_request'
env:
GH_TOKEN: ${{ github.token }}
REPO: ${{ github.repository }}
run: |
id=$(gh api "repos/$REPO/actions/artifacts?name=backend-coverage-main&per_page=20" --jq '
[.artifacts[]
| select(.expired | not)
| select(.workflow_run.head_branch == "main")
| select(.workflow_run.head_repository_id == .workflow_run.repository_id)
][0].id // empty')
if [ -z "$id" ]; then
echo "No main coverage artifact; reporting without a comparison."
exit 0
fi
gh api "repos/$REPO/actions/artifacts/$id/zip" > coverage-base.zip
unzip -q coverage-base.zip -d coverage-base
echo "found=true" >> "$GITHUB_OUTPUT"
- name: Report coverage on PR
if: github.event_name == 'pull_request'
uses: davelosert/vitest-coverage-report-action@v2
with:
json-summary-path: src/backend/coverage/coverage-summary.json
json-final-path: src/backend/coverage/coverage-final.json
json-summary-compare-path: coverage-base/coverage-summary.json
json-summary-compare-path: ${{ steps.base.outputs.found == 'true' && 'coverage-base/coverage-summary.json' || '' }}
+1 -1
View File
@@ -16,7 +16,7 @@ jobs:
- name: Set up Node.js
uses: actions/setup-node@v7
with:
node-version: '24'
node-version: '26'
cache: 'npm'
- name: Install dependencies
+3 -3
View File
@@ -57,7 +57,7 @@ jobs:
- name: Set up Node.js
uses: actions/setup-node@v7
with:
node-version: '24'
node-version: '26'
cache: 'npm'
- name: Install dependencies
@@ -78,7 +78,7 @@ jobs:
- name: Set up Node.js
uses: actions/setup-node@v7
with:
node-version: '24'
node-version: '26'
cache: 'npm'
- name: Install dependencies
@@ -135,7 +135,7 @@ jobs:
if: needs.changes.outputs.puterjs == 'true'
uses: actions/setup-node@v7
with:
node-version: '24'
node-version: '26'
cache: 'npm'
- name: Install dependencies
+1
View File
@@ -66,6 +66,7 @@ AGENTS.md
coverage/
.vitest-reports/
*.log
undefined
servers.json
+98 -76
View File
@@ -1,18 +1,18 @@
# AGENTS.md
Guidance for AI coding agents working in this repository. Read this before making changes. FOLLOW GUIDANCE AS CLOSELY AS POSSIBLE. If you think the guidance is wrong, raise an issue or flag a maintainer — don't just do what you think is right. This is the source of truth for how we write code, tests, and docs in this repo.
Guidance for AI coding agents working in this repository, and the source of truth for how we write code, tests, and docs here. Read it before making changes. FOLLOW GUIDANCE AS CLOSELY AS POSSIBLE. If you think a rule is wrong, raise an issue or flag a maintainer instead of doing what you think is right.
## Documentation Index
Use these as the source of truth before exploring further:
- [README.md](README.md) — project overview and quickstart.
- [doc/architecture.md](doc/architecture.md) — backend layered stack (controllers → drivers → services → stores → clients), `PuterServer` wiring, `Context` (ALS), and extensions.
- [doc/architecture.md](doc/architecture.md) — backend layered stack (controllers → drivers → services → stores → clients), `PuterServer` wiring, `Context` (ALS), actors, and extensions.
- [doc/contributing-apis.md](doc/contributing-apis.md) — adding and maintaining public APIs end to end (backend surface → puter.js → types → docs → tests). Follow it for any API work.
- [doc/pagination.md](doc/pagination.md) — the one pagination convention for list APIs (limit/cursor/offset/includeTotal, envelope shape, cursor semantics).
- [doc/alarms.md](doc/alarms.md) — raising alarms and picking a severity (what pages, what only gets recorded), plus the config that routes them.
- [doc/self-hosting.md](doc/self-hosting.md) — running Puter outside hosted infra.
- [CONTRIBUTING.md](CONTRIBUTING.md) — testing, security, AI-assisted code, PR conventions, Boy Scout Rule.
- [CONTRIBUTING.md](CONTRIBUTING.md) — contributor entry point and PR conventions.
- [SECURITY.md](SECURITY.md) — how to report vulnerabilities (do not file them publicly).
- [BUG-BOUNTY.md](BUG-BOUNTY.md) — bounty program scope.
- [TRADEMARK.md](TRADEMARK.md) — trademark usage.
@@ -26,16 +26,16 @@ These apply everywhere — backend, puter.js, and GUI.
### Language & files
- Write ES modules, not CommonJS — we transpile and build as needed.
- TypeScript preferred for new files in the backend and extensions; convert existing JS there opportunistically when you're already touching a file. The GUI and puter.js are plain JavaScript — don't introduce TypeScript files in them. In puter.js, type with JSDoc instead (see the puter.js section for what must be typed).
- **Reuse before adding.** Search for an existing type, helper, or implementation and extend it; only add a new one when nothing suitable exists.
- Make new types findable: descriptive `PascalCase` names, exported from the obvious entry point. A type used from many places belongs in the owning module's `types.ts` (e.g. [src/backend/controllers/types.ts](src/backend/controllers/types.ts)) rather than bloating a logic file; a type with a single consumer can stay next to it.
- Naming: `camelCase` for variables/functions, `PascalCase` for classes and files containing a class (`AuthService.ts`). **Prefer `camelCase` for all new code** — new local variables, parameters, functions, and internal object properties. `snake_case` is reserved for identifiers that are part of an external contract: wire/JSON keys sent to or received from the backend, stable API error codes, public option names that already ship as `snake_case`, and established `puter_*` namespaced fields. Don't rename those (it's a breaking change), but don't introduce new `snake_case` alongside them either — a camelCase local can carry a value into a `snake_case` wire key (`{ operation_id: operationId }`).
- TypeScript preferred for new files in the backend and extensions; convert existing JS there opportunistically when you're already touching a file. The GUI and puter.js are plain JavaScript — don't introduce TypeScript files in them.
- In plain-JS files, typing with JSDoc (`@type`/`@param`/`@returns`, `@typedef` for shared shapes, using the TypeScript type system) is encouraged. API types must not be `unknown` or untyped `...args`: spell out the real parameter and return shapes, unless the value passes through transparently to an upstream layer that owns its type. puter.js has stricter rules ([below](#types)).
- Make new types findable: descriptive `PascalCase` names, exported from the obvious entry point. A type used from many places goes in the owning module's `types.ts` (e.g. [src/backend/controllers/types.ts](src/backend/controllers/types.ts)); a type with a single consumer can stay next to it.
- Naming: `camelCase` for variables and functions, `PascalCase` for classes and files containing a class (`AuthService.ts`). **Prefer `camelCase` for all new code**, including parameters and internal object properties. `snake_case` is reserved for external contracts that already use it: wire/JSON keys, stable API error codes, public option names that already ship as `snake_case`, and established `puter_*` namespaced fields. Don't rename those (it's a breaking change), and don't add new `snake_case` beside them; a camelCase local can carry a value into a wire key (`{ operation_id: operationId }`).
### Comments
Keep comments light; prefer self-documenting code. Comment only when the _why_ is non-obvious or a usage detail would trip the next reader. Use `//` for single lines and `/** ... */` JSDoc when it genuinely needs more — if a comment runs long, it's probably too long. Don't restate the code and don't reference the current task, PR, or version — those rot. **No ticket references** (`PUT-1234`, `// fix for FOO-99`) in code, comments, or test names; describe the why in domain terms, not project-management terms. Use plain ASCII `-` in comment section dividers (`// -- Section --`), never box-drawing characters.
Prefer self-documenting code. Comment only when the _why_ is non-obvious or a usage detail would trip the next reader. **A comment is a sentence or two**: one or two lines of `//`, a few of `/** ... */` JSDoc. State the constraint and stop. Don't restate the code, narrate history ("used to swallow the failure, which meant…"), argue the design, or explain parameters the signature already shows. Reasoning that needs paragraphs belongs in a design doc.
**Be concise — a comment is a sentence or two, not a paragraph block.** One or two lines for `//`, a few for JSDoc. Don't narrate the history ("used to swallow the failure, which meant…"), don't argue the design at length, and don't explain each parameter when the signature already does. State the constraint and stop. If the reasoning genuinely needs paragraphs, it belongs in a design doc, not above the function.
Don't reference the current task, PR, or version; those rot. **No ticket references** (`PUT-1234`, `// fix for FOO-99`) in code, comments, or test names: describe the why in domain terms. Section dividers use plain ASCII (`// -- Section --`), never box-drawing characters.
### Security & privacy
@@ -47,77 +47,59 @@ Before opening a PR, scan the diff for:
When in doubt, return less. Auth-, permission-, or data-export-related changes deserve an explicit callout in the PR description.
Commit messages and PR descriptions never credit anyone or mention a bug report. For security fixes, state what the change does mechanically ("scoped tokens do xyz now"), not the vulnerability or how it could be abused.
### Reuse and abstraction
- **Search before you build.** Before writing a type, helper, cache/TTL map, retry, lock, pagination, validator, serializer, error mapper, host/email normalizer, or provider adapter, grep `src/backend/util/`, the owning layer, and `extensions/` for one that exists, and extend it instead of adding a parallel copy. If you find two copies, consolidate them or flag it. Before opening the PR, check the diff for new helpers that duplicate existing ones.
- **One path per operation.** Single-item variants call the batch version. Legacy and v2 endpoints share one service method. A second entry point (OIDC, WebDAV, admin, cron) calls the same service as the first instead of re-implementing it.
- **No abstraction without a second caller.** No base class, interface, factory, registry, decorator, or option bag with one implementation. No wrappers that only forward, no interface methods nobody calls, no methods only tests call.
- **Variants are data.** Providers or models that differ only in constants (base URL, prices, key names) are entries in a table read by one implementation, not one file each.
### Hot paths
- No per-row queries or awaits in loops: use the store's batch method (add one if missing) or [src/backend/util/concurrency.ts](src/backend/util/concurrency.ts).
- Bound everything: queries get `LIMIT`/pagination, in-memory maps get eviction, outbound calls get a timeout and a size cap.
- Build clients, catalogs, and config-derived lookups once, not per request.
- Listeners on fleet-wide events (`fs.write.file`, route lifecycle) return before any I/O for requests they don't concern.
### Working rules of thumb
- **Run it, don't just compile it.** "It type-checks" is not "it works." Exercise the code path end-to-end at least once.
- **Read the neighbors before writing.** Match the shape of similar things already in the tree. If you think the existing pattern is wrong, raise it — don't quietly diverge.
- **Test new behavior.** Every new function, endpoint, driver method, or logic branch gets a test; every bug fix gets a regression test that fails before the fix. If something is genuinely hard to test, skip it but say so in the PR.
- **Boy Scout Rule, proportional to the change.** Fix the obvious typo or dead import in files you're already touching; don't ride a refactor along with a bug fix.
- **Understand what you commit.** AI assistance is fine; shipping code you couldn't defend in review is not.
- **Boy Scout Rule, proportional to the change.** In files you're already touching, fix the typo, the dead import, the missing test, the bit you had to read twice. Don't ride a refactor along with a bug fix.
- **Understand what you commit.** AI assistance is fine; shipping code you couldn't have written, debugged, or defended in review is not.
- **A limit change is not done until the docs change with it.** Every rate limit, concurrency cap, quota, and allowance in the code is published in [src/docs/src/rate-limits-and-quotas.md](src/docs/src/rate-limits-and-quotas.md) — an undisclosed limit is one developers discover as a service failure. If a PR moves any of these numbers, the same PR updates that page. The numbers live in `src/backend/controllers/fs/limits.ts` (filesystem), `src/backend/drivers/util/aiLimits.ts` (all AI drivers), per-driver `rateLimit`/`concurrent` configs, and `src/backend/data/subPolicies/` (free-tier allowances).
---
## Local development configuration
## Local development
To test a local Docker build, create `docker-compose.override.yml` in the repository
root (or merge into the existing file):
To test a local Docker build, use the gitignored `docker-compose.override.yml` described in [Self-Hosting → Building from source](doc/self-hosting.md#building-from-source-instead-of-pulling). Keep local build settings there, not in `docker-compose.yml`.
```yaml
services:
puter:
pull_policy: never
build:
context: .
```
Compose merges it automatically. Run `docker compose up -d --build` to build and
start the stack, and repeat after source changes. Keep local build settings in
this gitignored override instead of editing `docker-compose.yml`, so they stay
out of pull requests and do not conflict when pulling updates. See
[Self-Hosting Puter](doc/self-hosting.md#building-from-source-instead-of-pulling)
for the required runtime configuration.
For local API testing, agents may decide whether to set `"unlimitedMetering": true`
when budget checks block unrelated tests; no additional approval is needed.
Merge it into the ignored `config.json` for `npm start` or
`puter/config/config.json` for Docker, then restart Puter. It applies to all
accounts on that local instance, including guests, and usage is still recorded.
Keep normal settings when testing budget or subscription enforcement, and leave
shared defaults unchanged.
When budget checks block unrelated local API tests, you may set `"unlimitedMetering": true` without further approval. Merge it into the gitignored `config.json` (`npm start`) or `puter/config/config.json` (Docker) and restart Puter. It covers every account on that instance, guests included, and usage is still recorded. Keep normal settings when testing budget or subscription enforcement, and leave shared defaults unchanged.
---
## Backend
A layered stack with explicit dependency injection: each layer depends only on the layers beneath it, receives them through its constructor, and `PuterServer` ([src/backend/server.ts](src/backend/server.ts)) wires the whole thing together. [doc/architecture.md](doc/architecture.md) is the full reference.
Controllers → drivers (optional) → services → stores → clients, all configured from `IConfig`. Each layer depends only on the layers beneath it and receives them through its constructor; `PuterServer` ([src/backend/server.ts](src/backend/server.ts)) wires them. [doc/architecture.md](doc/architecture.md) is the full reference.
| Layer (top → bottom) | Lives in | Responsibility |
| -------------------- | ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Controllers** | [src/backend/controllers/](src/backend/controllers/) | Route handlers: parse/validate input, per-route gates via `RouteOptions` (auth, subdomain, rate limit, body parsers), call services, format responses. |
| **Drivers** | [src/backend/drivers/](src/backend/drivers/) | Optional RPC-style handlers on `/drivers/*`; thin shells that validate inputs and call services/stores. |
| **Services** | [src/backend/services/](src/backend/services/) | Business logic. Assume the caller is already authenticated/authorized. |
| **Stores** | [src/backend/stores/](src/backend/stores/) | Persistence; wraps clients in the domain shapes services consume. |
| **Clients** | [src/backend/clients/](src/backend/clients/) | Adapters for external/internal services (sql, redis, s3, email, event bus). Protocols, not domain concepts. |
| **Config** | `config.*.json` → `IConfig` | Flat, typed config object every layer receives at construction. |
A public API can be a controller or a driver. Prefer a controller when you need fine-grained control over routes and gates; [doc/contributing-apis.md](doc/contributing-apis.md#controller-or-driver) has the decision guide.
A public API can be exposed through either a controller or a driver — both are supported; prefer a controller when you need fine-grained control over routes and gates. [doc/contributing-apis.md](doc/contributing-apis.md) has the decision guide with links to the decorators and middleware for each.
Layer rules:
Cross-layer rules:
- **Don't reach across layers.** Controllers don't poke clients; services don't register routes. If you want to, the abstraction is wrong — fix the abstraction.
- **Don't reach across layers.** Controllers don't poke clients; services don't register routes; lower layers never import from controllers. If you want to, the abstraction is wrong; fix the abstraction.
- **Gates run at the edge.** Controllers and drivers validate input and apply auth gates; services assume the caller is already authenticated and authorized.
- **Don't call sideways within a layer for code reuse.** Two services needing the same logic means a util/helper, not a service-to-service dependency.
- **Prefer explicit arguments over `Context` (ALS).** Reach for [Context](src/backend/core/context.ts) only for genuinely request-scoped values that would otherwise thread through many layers — today mostly `actor` and `req`.
- **A table's store owns every query against it.** Services, controllers, and extensions don't run raw SQL/redis against a table that has a store; add the store method so cache invalidation stays with the write.
- **Drivers and controllers stay thin.** Domain rules a second entry point needs (signup, app CRUD, worker deploy, AI routing/billing) live in a service, not in the first driver or controller that needed them.
- **Prefer explicit arguments over [`Context`](src/backend/core/context.ts) (ALS).** Use it only for request-scoped values that would otherwise thread through many layers; today that's mostly `actor` and `req`.
### Extensions
[extensions/](extensions/) parallels the layered stack and is for **non-crucial parts of the system** — Puter still works with the extension removed. If core needs to call it, it belongs in core, not an extension (see [whoami](extensions/whoami.ts) as the cautionary example). The `extension` global ([src/backend/extensions.ts](src/backend/extensions.ts)) exposes:
- `extension.registerClient/Store/Service/Driver/Controller(name, Class)` for first-class additions.
- `extension.on(event, handler)` and `extension.get/post/put/delete/patch/use(path, opts?, handler)` for the lightweight common case. `opts` is the same `RouteOptions` controllers use.
- `extension.import('client' | 'store' | 'service' | 'controller' | 'driver')` — lazy proxy to instantiated core objects. `extension.config` exposes live config.
Follow the same layered structure inside an extension — unless it only needs a few route handlers, in which case the lightweight helpers are enough on their own.
[extensions/](extensions/) is for **non-crucial parts of the system**: Puter must still work with the extension removed. If core needs to call it, it belongs in core ([whoami](extensions/whoami.ts) is the cautionary example). Inside an extension, follow the same layered structure, unless it only needs a few route handlers, in which case the `extension.get/post/...` helpers are enough on their own. The `extension` API is in [doc/architecture.md](doc/architecture.md#extension-api).
### Backend tests
@@ -130,44 +112,84 @@ Follow the same layered structure inside an extension — unless it only needs a
## puter.js (the SDK)
[src/puter-js/](src/puter-js/) is the public SDK. It ships live from `https://js.puter.com/v2/` with no version pinning — every existing app picks up changes immediately. Treat every observable behavior (signatures, response fields, error codes) as something a production app depends on.
[src/puter-js/](src/puter-js/) is the public SDK. It ships live from `https://js.puter.com/v2/` with no version pinning, so every change reaches every existing app at once; see [Rule zero](doc/contributing-apis.md#rule-zero-dont-break-callers).
Layout: SDK modules in [src/puter-js/src/modules/](src/puter-js/src/modules/) (one file or directory per area — `FileSystem/`, `kv/`, `ai/`, …), shared plumbing in [src/puter-js/src/lib/](src/puter-js/src/lib/), generated (gitignored) type declarations in `src/puter-js/types/`, API tests in [src/puter-js/tests/api/](src/puter-js/tests/api/), UI e2e tests in [src/puter-js/tests/e2e/](src/puter-js/tests/e2e/), developer docs in [src/docs/](src/docs/).
Language & typing: puter.js source is plain JavaScript — never TypeScript files — typed via JSDoc. **The JSDoc is the source of truth for the SDK's types.** `src/puter-js/types/` is `tsc --emitDeclarationOnly` output: the SDK build (`npm run build` in `src/puter-js`) generates it, the npm tarball ships it so TypeScript consumers get declarations, and git ignores it — it is never committed and never edited by hand. `npm run check:puterjs:types` generates it in CI and type-checks the result without `skipLibCheck`. The one hand-written declaration file is [src/puter-js/index.d.ts](src/puter-js/index.d.ts), which decides what is public and re-exports the generated names.
Every SDK change carries all five of these, or the PR is incomplete. [doc/contributing-apis.md](doc/contributing-apis.md) has the steps for each.
Declare a shape where it belongs, and reference it with an `import(...)` type from elsewhere. A shape more than one file needs lives in the module's `types.js`; a shape shared across modules lives in [src/puter-js/src/lib/types.js](src/puter-js/src/lib/types.js); a shape with one consumer can stay next to it:
1. **Backward compatibility**, unless a maintainer explicitly signs off on a break ([rules](doc/contributing-apis.md#maintaining-an-existing-api)).
2. **Tests** in the cross-platform API suite, or the Playwright harness for desktop-rendered `puter.ui.*` ([steps](doc/contributing-apis.md#6-tests)).
3. **Docs** in [src/docs/src/](src/docs/src/) that match the implementation exactly ([steps](doc/contributing-apis.md#5-docs), [writing rules](#developer-docs)).
4. **Types**, per the rules below.
5. **Error handling**: `{ message, code }` with stable `snake_case` codes, as in [modules/kv/](src/puter-js/src/modules/kv/) ([steps](doc/contributing-apis.md#3-puterjs)).
### Types
puter.js source is plain JavaScript typed via JSDoc, and **the JSDoc is the source of truth for the SDK's types**. `src/puter-js/types/` is `tsc --emitDeclarationOnly` output: the SDK build (`npm run build` in `src/puter-js`) generates it, the npm tarball ships it, and git ignores it. Never commit or hand-edit it. `npm run check:puterjs:types` generates it and type-checks the result without `skipLibCheck`; run it after any type change. The one hand-written declaration file is [src/puter-js/index.d.ts](src/puter-js/index.d.ts): it decides what is public and re-exports the generated names, so name a new type there if consumers should be able to import it.
- Public (exposed) methods carry JSDoc types for parameters and return value, with one `@overload` block per accepted call form; those overloads _are_ the published signature. Typing private helpers is at your discretion; annotate where it helps the next reader.
- Tag members `@internal`, not `@private`, to strip them from the generated declarations.
- Declarations must match runtime behavior exactly. A wrong type is worse than a missing one.
Declare a shape where it belongs and reference it elsewhere with an `import(...)` type. A shape more than one file needs lives in the module's `types.js`; one shared across modules lives in [src/puter-js/src/lib/types.js](src/puter-js/src/lib/types.js); one with a single consumer can stay next to it:
```js
/** @typedef {import('./types.js').KVOptConfig} KVOptConfig */
```
Use `@typedef {Object}` + `@property` for any shape whose fields need documenting — it is the only JSDoc form that carries a doc comment per field into the generated declaration. Keep the inline object-literal form for small internal shapes with nothing to say about each field, and prefer `unknown` over `*`:
Use `@typedef {Object}` + `@property` for any shape whose fields need documenting; it's the only JSDoc form that carries a per-field doc comment into the generated declarations. Use the inline form for small internal shapes with nothing to say per field, and prefer `unknown` over `*`:
```js
/** @typedef {{ key: string; value: unknown }} KVEntry */
```
Public (exposed) methods must carry JSDoc types — parameters and return value — with one `@overload` block per accepted call form, since those overloads _are_ the published signature. Unexposed/private helpers are typed at the contributor's discretion: annotate where it helps the next reader, and either way keep them clean. Members tagged `@internal` are stripped from the generated declarations, so use that tag rather than `@private` to keep something off the public surface.
---
Typing in JS files is encouraged: annotate with JSDoc `@type`/`@param`/`@returns` using the TypeScript type system, and define shared shapes with `@typedef`. API types must not be `unknown` or untyped `...args` — spell out the real parameter and return shapes; the only exception is values passed through transparently to an upstream layer that owns their type. For example:
## Developer docs
```js
/** @typedef {{ key: string; value: unknown }} KVEntry */
[src/docs/src/](src/docs/src/) is the published puter.js documentation: one reference page per method (`<Area>/<method>.md`, [steps](doc/contributing-apis.md#5-docs)) and recipes in [src/docs/src/recipes/](src/docs/src/recipes/).
/** @type {KVEntry[]} */
let entries = [];
```
### Page structure
Every SDK change carries all five of the following — a puter.js PR missing one is incomplete:
The reference has two kinds of page:
1. **Backward compatibility.** Mandatory unless a maintainer explicitly signs off on a break. Existing call signatures keep working (including both positional and options-object forms where a method supports them); new parameters are optional with defaults that preserve old behavior; never rename or repurpose existing params, response fields, or error codes. New parameter names are `camelCase` (existing `snake_case` stays for compatibility). Say in the PR how existing callers are unaffected.
2. **Tests.** Add or extend a suite in [tests/api/suites/](src/puter-js/tests/api/suites/) (`<area>.suite.ts`; register new suites in `suites/index.ts` — no globbing). One suite runs unchanged on node, browser, and workerd via `npm run test:puterjs`; never write a per-platform test. The runners execute the **built** bundle — run `npm run build:workerLib` after SDK changes or the suite silently tests stale code. For `puter.ui.*` methods rendered by the desktop, use the Playwright e2e harness instead — see [src/puter-js/TESTING.md](src/puter-js/TESTING.md).
3. **Docs.** New or changed APIs update [src/docs/src/](src/docs/src/): the method page (`<Area>/<method>.md`, with frontmatter and a runnable example) and the area overview when the surface changes. Docs are the contract users code against — signatures, defaults, and return shapes must match the implementation exactly.
4. **Types.** Type the change in JSDoc on the implementation, then run `npm run check:puterjs:types` to confirm it still produces declarations that type-check. Name any new type in [src/puter-js/index.d.ts](src/puter-js/index.d.ts) if consumers should be able to import it. Declarations must match runtime behavior exactly — a wrong type is worse than a missing one.
5. **Error handling.** Reject/throw `{ message, code }` objects with stable `snake_case` codes, matching the existing modules (see `KV.js`). Validate cheap preconditions client-side before making the network call; pass backend errors through unchanged rather than swallowing or re-wrapping them. Error codes are API surface — changing one is a breaking change.
- **API overview pages** (`<Area>.md`, such as [KV.md](src/docs/src/KV.md)) give a quick top-level picture of what the API is for and what it can do, with short examples and a list of its functions that links every function page in the area. Leave out the details. The first sentence of the intro says what the API is about. This is a judgment call too: when the API depends on a Puter concept the reader may not know, such as Teams, the first sentence can name it and the sentences after it explain it.
- **Function pages** (`<Area>/<method>.md`) hold the details: parameters, defaults, return values, errors, limits, and edge cases.
[doc/contributing-apis.md](doc/contributing-apis.md) walks the full lifecycle of adding an API across backend + SDK.
An overview page answers "what is this API and is it what I need?", and a function page answers "how exactly does this call behave?". A change to one function usually only touches that function's page, so a new `kv.list()` option goes in `KV/list.md` without a matching update to `KV.md`. Don't copy a function's options, limits, or edge cases onto the overview page for completeness. Where a detail goes is a judgment call, not a hard rule: update the overview when a change affects how someone would describe the API as a whole.
When you add a page, add it to [src/docs/src/sidebar.js](src/docs/src/sidebar.js) under its area, or it won't show up in the navigation. When you add a runnable example, add its HTML to `src/docs/src/playground/examples/<slug>.html`, register the slug in [src/docs/src/examples.js](src/docs/src/examples.js), and link the code block to it by putting the slug after the language with a semicolon (` ```html;kv-set `).
### Writing docs
These apply to reference pages and recipes alike.
- **Document the developer-facing side only.** The reader is building on top of puter.js, so cover what affects their code: signatures, defaults, return shapes, errors, limits, and behavior they can observe. Leave out internal workings (which service handles the call, storage layout, caching, retries) unless they change what the developer has to do. [rate-limits-and-quotas.md](src/docs/src/rate-limits-and-quotas.md) is the exception: it publishes every limit, including timeouts and caps that come from how the backend works.
- **Describe the current state.** Docs aren't a changelog. Write how the API behaves now, without "now", "no longer", "existing code is unaffected", or other wording that compares it to an older version.
- **Verify before you document.** Run the calls you describe against a running Puter instead of inferring behavior from the source.
- **Write plainly.** Use the literal words a developer would say out loud, with contractions and everyday words. No idioms or figurative verbs ("walk away from something shared with you" → "remove your own access", "nothing to page through" → "without pagination"). No formal register ("present only while more pages remain" → "includes a `cursor` if there are more pages to load"). Prefer dev terms such as atomic, concurrent, and insertion order.
- **State the rule, then its consequence, in short sentences.** No metaphor in place of a rule. Name a concrete action ("you can create any files and folders inside it") over an abstract permission ("yours to organize freely"). Cut a sentence that only repeats the next one.
- **Describe behavior directly.** No quoted inner-voice phrases. Write "`isNew` is `true` when the person did not have access before", not "this tells 'I just gave this to someone' from 'they already had it'".
- **Keep references easy to follow.** Name what a list or code block belongs to in the sentence directly above it. Name a thing once and keep that name. Introduce an actor as "a user" before giving them a role: "A user can create a team. The user who creates the team becomes its owner."
- **Sentence mechanics.** Never start a sentence with inline code: write "To store data, use the [`set()`](/KV/set/) method." Link every inline mention of an API to its page, every time. No em dashes, and no colons or semicolons mid-sentence; a colon that introduces a code block is fine.
- **Show the correct way.** No "don't" and no wrong-way-first examples, in recipes and playground examples alike. Mention a wrong approach only when developers already hit it in practice, and state how it fails as a fact: "`add()` reads a plain object as a path map, so the call rejects with …".
### Recipes
Recipes are tested, copy-pasteable patterns for a task, one layer above the per-method reference, often combining several APIs. Agents find them through the `## Recipes` section of `llms.txt`.
- **Files.** One `.md` per recipe in [src/docs/src/recipes/](src/docs/src/recipes/), with `title`, `description`, `tags`, and optional `order` front matter; they're picked up automatically. Tags must exist in [src/docs/src/recipe-tags.js](src/docs/src/recipe-tags.js) or the build fails. Quote a `description` that contains a colon followed by a space. Every top-level `.md` in that directory is built as a recipe, so keep other markdown out of it.
- **Title.** Short, Title Case, starts with a verb, and leaves out "to Your App" ("Add Team Support", "Deploy Workers Programmatically"). The slug is the title in kebab case, with no API-name prefix. In `llms.txt` an agent sees only the title and description before deciding to open a recipe, so word them to match the task the reader is trying to do.
- **Description.** Written for someone who has never heard of Puter. Start with "Learn how to", put the action first, and say what the reader's app gains: "Learn how to add file sharing to your app with Puter.js, so users can share files in their Puter account with other people." No API names, no mechanism, no list of example cases, and no "not just …" contrast.
- **Intro.** Explain the situation and why you'd do this, tied to something the reader already knows ("`puter.kv` is your database in Puter.js"). No "Use this when" bullet list.
- **Structure.** Get to the first working call as early as possible, then add validation, rate limits, and sign-in after it. Explain a Puter concept the first time it appears, such as what `me.puter` is or that `puter.workers.exec()` opens the Puter sign-in window. Split recipes by what the reader needs, not by technique, and give a family of recipes a shared tag.
- **Reading order.** A recipe reads top to bottom. Each code block uses only functions and variables from blocks above it, so the reader never has to search further down the page to understand it. Start with a small version that works, and have each section add one case. When a section changes a function from earlier, show the whole new version of that function there. If the code is built up over several sections, end with a `## Full Code` block that has all of it.
- **Link to the reference for full option lists.** Mention only the options the recipe uses, then link to the function page for the rest ("All the sort options are listed on [`puter.fs.readdir()`](/FS/readdir/)."). A copied list of options, rules, or scopes goes stale when the reference changes.
- **Limits.** Mention a limit only when it's the reason for a pattern the recipe uses, such as one wildcard subscription instead of one per key, and link to [rate-limits-and-quotas.md](src/docs/src/rate-limits-and-quotas.md) or the function page for the number. Leave out limits that code following the recipe never reaches.
- **Code.** Platform-neutral: just the code, with top-level `await`. No `<script>` or `<html>` wrapper, no IIFE.
- **Tradeoffs.** Justify a choice with one scenario the reader can picture right away, in about two short paragraphs, or fold each point into the section where it matters. The description states the benefit only.
- **Notes.** Leave out `## Notes` by default. Use it only to point to an architecturally different approach. A limit the recipe needs goes in the section it applies to, and other facts belong on the linked reference page.
---
@@ -175,7 +197,7 @@ Every SDK change carries all five of the following — a puter.js PR missing one
[src/gui/](src/gui/) is the Puter desktop: deliberately plain JavaScript + jQuery with HTML-string templates. Don't introduce a UI framework or a new rendering pattern.
The guiding rule here is **conformity over novelty** — match the existing design and code structure even where you'd personally choose differently. A visually or structurally divergent addition is a defect even when it works.
**Conformity over novelty**: match the existing design and code structure even where you'd choose differently. A visually or structurally divergent addition is a defect even when it works.
- **Reuse existing UI primitives before writing new ones.** Windows and dialogs are `UIWindow*` functions in [src/gui/src/UI/](src/gui/src/UI/); generic pieces already exist (`UIAlert`, `UIPrompt`, `UIContextMenu`, `UINotification`, `UIPopover`, widgets in [UI/Components/](src/gui/src/UI/Components/)). A new window should read like its neighbors: an async function taking an options object, composing an HTML string, wiring behavior with jQuery, delegating to `UIWindow(...)`.
- **Match the visual language.** Use existing CSS classes (`button`, `button-primary`, window chrome, form styles) and copy the layout patterns of neighboring windows; new styles go in [src/gui/src/css/](src/gui/src/css/) following existing conventions. Verify anything positional (menus, overlays, z-index) on both desktop and mobile viewports.
+26 -8
View File
@@ -17,6 +17,21 @@ Out-of-scope:
- Denial of Service (DoS), spam, or volumetric attacks.
- Physical security issues.
## Eligibility
We pay bounties for issues that threaten user security and privacy, or the security and health of the platform. We appreciate every report, but hardening suggestions and correctness bugs are not eligible, even when we fix them. Please open a GitHub issue for those.
A report must show how an attacker can directly perform the action against another account or the system. It is not eligible if it:
- Starts from access the attacker should not already have: a stolen session or token, the victim's password, a root or full-access token, or script execution on `puter.com`.
- Needs the victim to do more than ordinary use (opening a link or an app), such as publishing their own secrets, sending a verification link to an address they don't control, or approving a prompt that accurately states what it grants.
- Gives the attacker nothing they couldn't already get through a supported API.
- Only reproduces in a non-default or development setup (SQLite, a `custom` OIDC provider, an unset outbound proxy, cookies or config planted by hand). Findings in the open-source code must affect `puter.com` or a default self-hosted install.
**Shared state.** Sharing files, KV entries, or cross-app KV access means the sharer trusts the sharee. Exploits that depend on that trust, such as a sharee misusing access the sharer granted, may be treated as hardening or lower severity.
Hardening we ship in a public PR does not make similar code paths vulnerabilities. They are judged by these same rules.
## Known Non-Issues (Please Check Before Submitting)
The following have already been reviewed and determined **not to be vulnerabilities**. Reports that only re-describe one of these are **not eligible for a reward** and will be closed as non-issues — even if they include new code references. Please review this list before submitting:
@@ -27,11 +42,12 @@ The following have already been reviewed and determined **not to be vulnerabilit
- **Apps invoking drivers, creating workers, or using KV.** Applications are intended to do this; worker permissions are scoped to the owning app. This is by design.
- **App metadata or app user-count "leaks".** This information is currently public by design. This covers **private apps** too: private apps are an unreleased, admin-only beta for paid listings in the Puter App Center. Their `index_url` and other metadata are not secret, the flag is not user-settable, and no production app relies on it. Reports that a private app's metadata is reachable through a sibling route (taskbar items, `/get-launch-apps`, `index_url` drift on the hosting gate, etc.) are normalization work, not vulnerabilities.
- **Worker hot-reload driven by a share recipient.** Sharing a worker's source file with write access is how you let a collaborator change that worker's code and have it deploy live. The redeploy running as the file owner, with the owner's worker credentials, is the intended behavior of that grant — the owner chose to hand out write access to executable code. If you do not want that, do not share the source file with write access.
- **Re-claiming a site or worker name its owner deleted.** A hosted site or worker name that its owner deletes is treated as abandoned by that owner. If one of their apps still points at it, whoever claims the name next serves that app's `index_url` and receives what the app is launched with, and a new app on the same origin may share the old one's identity. Guarding app identities against being recycled this way is ongoing hardening, not a security boundary, so new ways to re-claim an abandoned name, or to inherit an app's tokens or data through one, are not eligible.
- **Share-mode downgrades that do not cascade.** Demoting a delegate does not retroactively revoke the grants that delegate already issued downstream. Shares are independent grants once issued, not a live-evaluated chain. Revoking the downstream grant is a separate, supported action.
- **Concurrency / TOCTOU overspend of credits or storage quota.** Firing concurrent requests to exceed an AI credit hold or a storage quota by some margin is known and accepted; the per-account concurrency and rate-limit gates are the control we rely on, and the residual overage is bounded and billed. Report these only if you can demonstrate _unbounded_ overspend.
- **Metering, credit-hold, and quota inaccuracies.** Credit holds and quota checks are up-front estimates, and actual usage is billed afterwards. Overspend that is bounded and still billed is a correctness bug, including concurrent requests that exceed a hold or quota by some margin; the per-account concurrency and rate-limit gates are the control we rely on. Report these only if you can use a service without it being billed, or overspend without bound.
- **Existence-only oracles.** A response that tells "exists but forbidden" apart from "not found" for a path or ID the caller already supplied, without revealing anything else about it, is hardening.
- **Session and credential lifecycle around 2FA enrollment.** Enrolling in 2FA is not a credential-revocation event: existing authenticated sessions stay valid, and outstanding password-reset links stay valid. Likewise, account-management actions behind the `/user-protected/*` gate are password-gated by design, not TOTP-gated. A recovery code that stops being accepted fails _closed_ — it denies access rather than granting it, so it is a bug report, not a vulnerability.
- **Group permissions** (`/auth/revoke-user-group` and siblings). The group-permission surface is unfinished and not exposed anywhere in the product. Findings there are not reachable by any user.
- **Correctness bugs with no security boundary crossed.** For example, an API that reports success while silently leaving data behind. We want these reported — please open a GitHub issue — but they are not bounty-eligible.
- **Missing Subresource Integrity (SRI) on external scripts/styles.** Accepted as hardening, not as a vulnerability, absent a demonstrated compromise of the CDN in question.
- **General "token in a URL" / token-lifetime / token-storage designs** — signed directory URLs exposing children, a write signature implying read, app tokens outliving a web session, or the session token being held in `localStorage` on `puter.com`. These are current intended behaviors. Shorter-lived tokens with a refresh flow are on the roadmap; their absence is not a vulnerability on its own. Show a reproducible attack that does not already presuppose script execution on `puter.com`.
- **Missing PKCE, unverified `id_token` signatures, or other OIDC hardening.** Provider tokens are obtained through a server-to-server authorization-code exchange with the provider's token endpoint over TLS, so the resulting `id_token` / userinfo claims are trusted from that channel rather than from local JWKS signature verification — the callback never accepts a caller-supplied `id_token`. Adding local signature / `aud` / `iss` / `exp` checks is welcome defense-in-depth (please open a GitHub issue/PR), but their absence is not an account-takeover vector on its own.
@@ -52,7 +68,7 @@ To participate, you must:
3. **Respect availability**: Do not perform denial-of-service attacks or automated scans that degrade service.
4. **Follow disclosure policy**: Do not publicly disclose vulnerabilities until we have confirmed and patched the issue.
5. **Act in good faith**: Make every effort to avoid privacy violations, destruction of data, and interruption or degradation of services.
6. **Check the Known Non-Issues list**: Reports matching an item in the "Known Non-Issues" section above are not eligible and will be closed.
6. **Check Eligibility and Known Non-Issues**: Reports that fall outside "Eligibility" or match an item in "Known Non-Issues" above are not eligible and will be closed.
Reports that do not meet these guidelines may not be eligible for a reward.
@@ -63,14 +79,14 @@ Reports **MUST** demonstrate a **working, reproducible exploit with real impact*
- Exact steps to reproduce, the relevant request/response or code path, and the commit or version you tested.
- The **observed** result versus the **expected** result.
- For client-side, UI, or authentication-flow bugs: a short screen recording (≤ 2 minutes) showing the exploit working end-to-end on a real Puter instance.
- For server-side bugs: a runnable proof-of-concept and screenshot or video showing the attack process.
- For server-side bugs: a runnable proof-of-concept and screenshot or video showing the attack process. A unit test or in-process harness alone does not count; show it against a running instance.
Reports based solely on reading the source ("source review only, not tested") or unverified AI/LLM-generated reports are the **lowest triage priority and not eligible for bounty**.
If you used an AI tool to help find an issue, you must personally verify that it actually reproduces before submitting.
If you used an AI tool to help find an issue, you must personally verify that it actually reproduces before submitting. Reports sent by automated agents are not eligible.
Please submit **one issue per report**.
Bundled "audit packs" of many speculative findings will not be elligible for bounty; send each confirmed issue separately.
Additionally please **DO NOT** submit too many issues at once; these will be treated as a bundled report, and will not be elligible for bounty.
Bundled "audit packs" of many speculative findings will not be eligible for bounty; send each confirmed issue separately.
Additionally please **DO NOT** submit too many issues at once; these will be treated as a bundled report, and will not be eligible for bounty.
## Reporting Process
@@ -83,6 +99,8 @@ Include:
- Suggested remediation (if available)
- Paypal id where we can send the bounty (if eligible)
Send the full report in your first email. We don't reply to messages that only ask whether we run a bounty or where to report.
We aim to acknowledge receipt within **72 hours** and provide a resolution timeline.
## Reward Structure
@@ -95,7 +113,7 @@ We offer monetary rewards based on the severity of the vulnerability, as determi
- **Low: \$50 – \$100**
Non-security issues, suggestions, and best practices feedback are always welcome, but may not qualify for a reward.
If multiple researchers report the same issue, the bounty will be awarded to the first eligible report we receive.
If multiple researchers report the same issue, the bounty will be awarded to the first eligible report we receive. Issues we already know about, from internal review, an open ticket, or a fix already in a public commit or PR, count as duplicates.
## Payments Disclaimer
+11 -36
View File
@@ -1,46 +1,21 @@
# Contributing to Puter Backend
# Contributing to Puter
Thanks for contributing. These rules aren't strictly enforced — but following them makes every PR easier. If anything's unclear, ping a core maintainer or open the PR and ask.
Thanks for contributing. These rules aren't strictly enforced, but following them makes every PR easier. If anything's unclear, ping a core maintainer or open the PR and ask.
New to the backend? Start with [doc/architecture.md](doc/architecture.md).
- New to the backend? Start with [doc/architecture.md](doc/architecture.md).
- Code, test, and docs conventions for the whole repo live in [AGENTS.md](AGENTS.md). It's written for AI agents; the conventions apply to human contributors too.
- Adding or changing a public API (an endpoint, driver method, or puter.js method)? Follow [doc/contributing-apis.md](doc/contributing-apis.md): backward compatibility, [developer docs](src/docs/), types, and tests all move in the same PR.
- Found a vulnerability? Report it privately per [SECURITY.md](SECURITY.md).
---
## 1. Test it. Run it.
## Before you open a PR
Run the affected code path end-to-end before opening a PR. "It builds" is not "it works."
Check your diff against these sections of AGENTS.md:
Add tests for new behavior, endpoints, or bug fixes. If something's genuinely hard to test, say so in the PR.
## 2. Follow existing patterns
Match the shape of similar code already in the repo. [doc/architecture.md](doc/architecture.md) is the source of truth for layers, wiring, and naming. If you think a pattern is wrong, raise it — don't quietly diverge.
In plain-JS files, typing is encouraged via JSDoc `@type` annotations using the TypeScript type system, with `@typedef` for shared shapes. Don't type API surfaces as `unknown` or untyped `...args` unless the values are passed through transparently to an upstream layer that owns their type.
## 3. Don't expose system or user information
Scan your diff for stray logs, debug routes, internal paths, secrets, tokens, or user data in errors/responses. When in doubt, return less. Flag any auth, permission, or data-export changes in the PR description.
For private security reports, see [SECURITY.md](SECURITY.md).
## 4. AI-assisted code is fine — understood code is required
Don't commit code you couldn't have written, debugged, or defended yourself. Read the diff, run it, and be ready to explain it in review.
## 5. Adding or changing APIs
If you add or change a public API (an endpoint, driver method, or puter-js method), follow [doc/contributing-apis.md](doc/contributing-apis.md) — backward compatibility, [developer docs](src/docs/), types, and tests all move in the same PR.
The same rule applies to limits: rate limits, concurrency caps, quotas, and allowances are published in [src/docs/src/rate-limits-and-quotas.md](src/docs/src/rate-limits-and-quotas.md), and a PR that changes one of those numbers updates that page in the same PR. A limit nobody published is a limit developers only discover as a service failure.
## 6. Boy Scout Rule — leave it 1% better
![Boy Scout Rule](https://imgs.search.brave.com/DMmIWl5-NuZVtrR9kXBb06AKF8kturkgSW9UMb2-6m4/rs:fit:860:0:0:0/g:ce/aHR0cHM6Ly9sYXdz/b2Zzb2Z0d2FyZWVu/Z2luZWVyaW5nLmNv/bS9pbWFnZXMvbGF3/cy9ib3ktc2NvdXQt/cnVsZS5wbmc)
Fix the typo, the dead import, the missing test, the bit you had to read twice. Keep cleanup proportional to the change — no refactors riding along on bug fixes.
---
- [Working rules of thumb](AGENTS.md#working-rules-of-thumb): run it end to end, test new behavior and bug fixes, understand every line you commit (AI assistance is fine), leave touched files a bit better, and publish limit changes.
- [Reuse and abstraction](AGENTS.md#reuse-and-abstraction) and [hot paths](AGENTS.md#hot-paths).
- [Security & privacy](AGENTS.md#security--privacy): expose nothing the caller doesn't need.
## Opening a PR
+4 -2
View File
@@ -14,7 +14,7 @@
# config.default.json, so partial overrides work. Absent file = defaults.
# ---- Build stage ----
FROM node:24-slim AS build
FROM node:26-bookworm-slim AS build
WORKDIR /opt/puter
@@ -59,7 +59,7 @@ RUN set -e; \
wait $pjs_pid
# ---- Runtime stage (slim — no build tools) ----
FROM node:24-slim
FROM node:26-bookworm-slim
WORKDIR /opt/puter
@@ -79,6 +79,8 @@ ENV PUTER_CONFIG_PATH=/etc/puter/config.json
ENV NODE_OPTIONS=--enable-source-maps
EXPOSE 4100
# The inbound SMTP receiver, when run from this image with the `smtp` profile.
EXPOSE 2525
USER node
+1 -1
View File
@@ -24,7 +24,7 @@
"disable_user_signup": false,
"teams_enabled": false,
"max_teams_per_user": 1,
"max_seats_per_team_free": 4,
"max_seats_per_team_free": 3,
"max_seats_per_team_paid": 40,
"strict_email_verification_required": false,
"gui_assets_root": "./src/gui",
+55 -12
View File
@@ -7,9 +7,8 @@
// Each setting lives at exactly one canonical key — there are no fallback
// aliases. Values shown are illustrative, not production secrets.
//
// Keys consumed only by closed-source / hosted-prod extensions (clickhouse,
// cacheUpdateHandler, pages, prodMeteringAndBilling, …) are intentionally
// omitted.
// Keys consumed only by closed-source / hosted-prod extensions are
// intentionally omitted.
// ── Environment / identity ──────────────────────────────────────────
"config_name": "template",
@@ -35,7 +34,7 @@
// Idle keep-alive timeout in ms. Must stay above the idle timeout of any
// proxy in front, or the proxy reuses connections this server has closed
// and its clients see 502s.
"keep_alive_timeout": 620000,
"keep_alive_timeout": 920000,
"protocol": "http",
"domain": "puter.localhost",
// Fully-qualified externally-visible URL. Computed from protocol/domain/
@@ -216,7 +215,9 @@
// "bootstrapTables": true,
"path": "",
// Credentials. NOTE: snake_case here, unlike `s3.s3Config` below.
// For dynamodb-local, any non-empty values work.
// For dynamodb-local, any non-empty values work. On AWS,
// `"useCredentialChain": true` uses the default credential chain
// instead of keys.
"aws": {
"access_key": "",
"secret_key": "",
@@ -255,6 +256,38 @@
}
},
// ── Inbound mail (separate process) ─────────────────────────────────
// Mailboxes addressed at one of your own domains. `secret` gates the
// ingress endpoint that files a message into the recipient's ~/.mail.
//
// `localServer` additionally runs an SMTP listener that accepts mail from
// other mail servers and hands each message to that endpoint. It runs as
// its own process - `npm run start:smtp`, or the `smtp` compose profile -
// and stores nothing itself. Without it, nothing delivers to the endpoint
// unless you point something else at it.
//
// The listener is unauthenticated by nature: anything on the internet may
// hand it a message for an address in `localDomains`. It performs no spam,
// SPF, DKIM or DMARC checks. Read the self-hosting guide before exposing
// it.
// "userEmail": {
// "secret": "a long random string",
// "localServer": true,
// "localDomains": ["example.com"],
// // Container-internal by default; publish 25 onto it from outside.
// "localPort": 2525,
// // Defaults to <api_base_url>/email/ingress. The endpoint is served
// // on the `api` subdomain, so when this points somewhere internal,
// // `localIngressHost` has to name that virtual host.
// "localIngressUrl": "http://puter:4100/email/ingress",
// "localIngressHost": "api.example.com",
// "localHostname": "mx.example.com",
// "localMaxRecipients": 50,
// // Concurrent connections. Each message is held in memory while it
// // is delivered, so this also bounds memory use.
// "localMaxClients": 20
// },
// ── Sharing ─────────────────────────────────────────────────────────
// Email a recipient who already has an account about a new share. On
// unless set to false; they opt out via the unsubscribe link, or by
@@ -282,7 +315,8 @@
// The tables ship either way and sit inert; this gates whether `/teams`
// is registered at all, so with it off the paths do not exist rather than
// existing and refusing. It is also the backout: turning it off removes
// the feature without touching data.
// the feature without touching data. Teams ships as an extension; on,
// the server refuses to boot unless one registers it.
"teams_enabled": false,
//
// Live teams one user may own. Soft-deleted ones do not count, so
@@ -291,12 +325,13 @@
//
// Seats one team may provision. Each seat is a real Puter account taking a
// name from the global username pool, so this is what bounds a team's blast
// radius. The owner's plan decides which of the two applies; defaults are 4
// and 40. Lowering either below a team's current seat count blocks new
// provisioning and disables nobody. Deployments upgrading from the old flat
// default of 50: without a billing extension every owner reads as free, so
// set `max_seats_per_team` to keep the old behavior.
"max_seats_per_team_free": 4,
// radius. The owner's plan decides which of the two applies; defaults are 3
// and 40. A plan registered with `teamSeatCap` uses that instead. Lowering
// a cap below a team's current seat count blocks new provisioning and
// disables nobody. Deployments upgrading from the old flat default of 50:
// without a billing extension every owner reads as free, so set
// `max_seats_per_team` to keep the old behavior.
"max_seats_per_team_free": 3,
"max_seats_per_team_paid": 40,
//
// One flat cap for every team, whatever the owner pays. Setting it
@@ -398,6 +433,8 @@
"mistral": { "apiKey": "" },
"xai": { "apiKey": "" },
"moonshot": { "apiKey": "" },
// MiniMax. One key powers chat and video generation (H3, Hailuo);
// video calls go to the same host, beside the chat API's `/v1`.
"minimax": {
"apiKey": "",
"apiBaseUrl": "https://api.minimax.io/v1"
@@ -418,6 +455,10 @@
"apiKey": "",
"apiBaseUrl": "https://ark.ap-southeast.bytepluses.com/api/v3"
},
// Kling AI. One API key (not the legacy AccessKey/SecretKey pair)
// powers video generation, image generation and TTS; `apiBaseUrl`
// defaults to https://api-singapore.klingai.com.
"kling": { "apiKey": "" },
"zai": { "apiKey": "" },
"alibaba": { "apiKey": "" },
"hoonify": {
@@ -456,6 +497,8 @@
"defaultVoiceId": "",
"speechToSpeechModelId": ""
},
// Both AWS providers also take `"useCredentialChain": true` in place
// of keys, like `dynamo.aws`.
"aws-polly": {
"access_key": "",
"secret_key": "",
+9 -3
View File
@@ -36,9 +36,10 @@ pass one explicitly unless you really mean "page someone".
### Choosing one
- Did the server fail to do its job in a way nobody expected? → `critical`
- Is a background job, rate, or dependency degraded? → `warning`
- Is this a user doing something notable (tripping an abuse heuristic,
overspending)? → `info`
- Is a background job, rate, or dependency degraded, or has an account spent
far past its credits? → `warning`
- Is this a user doing something notable (tripping an abuse heuristic)?
→ `info`
- Did one of *our own* limits reject a caller — a rate limit, a concurrency
cap, a quota? → don't alarm at all. The limit doing its job is not an
event; the 429 is the whole signal, and alarming on it only produces noise
@@ -133,6 +134,11 @@ occurrence is reported up to a small burst, then one per interval. A fault
recurring hundreds of times a second still counts every occurrence, but it
does not write a log line or build an alert payload for each one.
The count lasts only as long as the client remembers the alarm. One that
hasn't recurred for 24 hours is forgotten, and the client holds at most 1000
at once, dropping the least recently raised first. Either way the next
occurrence is a new alarm, with its count and burst starting over.
Below that, the chat transport won't repost the same alarm id within
`repeatThrottleMs` (default 15 minutes). The first occurrence always posts,
and the next one that gets through reports how many piled up in between —
+9 -20
View File
@@ -1,6 +1,6 @@
# Backend Architecture
Loosely inspired by the Controller–Service–Repository pattern with dependency injection. The backend is organized as a stack of layers where each layer only depends on the layers beneath it, and `PuterServer` ([src/backend/server.ts](../src/backend/server.ts)) instantiates each layer in order and hands the instances down to the next.
Loosely inspired by the Controller–Service–Repository pattern with dependency injection. The backend is a stack of layers; each depends only on the layers beneath it and receives them through its constructor from `PuterServer` ([src/backend/server.ts](../src/backend/server.ts)), which instantiates the layers in order. The rules for working within the layers are in [AGENTS.md](../AGENTS.md#backend).
## Layers
@@ -31,8 +31,6 @@ block-beta
style CFG fill:#334155,stroke:#1e293b,color:#fff
```
Each layer only depends on the layers beneath it, and every dependency is injected through the constructor by `PuterServer`. Extensions sit alongside this stack and can register into any layer — see [Extensions](#extensions) below.
| Layer | Lives in | Responsibility |
| --- | --- | --- |
| **Controllers** | [src/backend/controllers/](../src/backend/controllers/) | Route handlers. Parse + validate input, apply per-route gates (auth, subdomain, rate limit, body parsers — see `RouteOptions`), call into services, format responses. |
@@ -42,7 +40,7 @@ Each layer only depends on the layers beneath it, and every dependency is inject
| **Clients** | [src/backend/clients/](../src/backend/clients/) | Adapters for external/internal services (sql, redis, s3, dynamodb, email, event bus, …). Knows protocols, not domain concepts. |
| **Config** | `config.*.json` → `IConfig` | The flat, typed config object every layer receives at construction. |
Each layer receives the layers beneath it through its constructor, so dependencies are explicit and traceable from `PuterServer`. A controller does not reach into a client directly; if it needs one, the right move is usually a service.
Extensions sit alongside this stack and can register into any layer; see [Extensions](#extensions).
## Entry point: `PuterServer`
@@ -55,13 +53,11 @@ Each layer receives the layers beneath it through its constructor, so dependenci
## Context (ALS)
We use [`Context`](../src/backend/core/context.ts) — backed by `AsyncLocalStorage` — to carry per-request state without threading it through every function signature. It is used **sparingly**, mostly for `actor` and `req`. The request-context middleware opens a scope per request after the auth probe runs; anything inside a request handler can call `Context.get('actor')` / `Context.get('req')` instead of plumbing it as an argument.
Prefer explicit arguments. Reach for `Context` only when the value is truly request-scoped and would otherwise need to thread through many layers.
[`Context`](../src/backend/core/context.ts), backed by `AsyncLocalStorage`, carries per-request state without threading it through every function signature. The request-context middleware opens a scope per request after the auth probe runs, so anything inside a request handler can call `Context.get('actor')` / `Context.get('req')`. It's used sparingly, mostly for `actor` and `req`; see the [Context rule](../AGENTS.md#backend).
## Actors
An [`Actor`](../src/backend/core/actor.ts) is who a request is acting as. It has two app-shaped fields and they are not interchangeable: `app` is the app the actor carries *directly* (an app-under-user token), while `effectiveApp` is the app it ultimately acts as — its own, or the one that issued its access token. An access-token actor has no `app` of its own, so a gate that reads `app` answers "no app" for it and falls open.
An [`Actor`](../src/backend/core/actor.ts) is who a request is acting as. Its two app fields are not interchangeable: `app` is the app the actor carries *directly* (an app-under-user token), while `effectiveApp` is the app it ultimately acts as — its own, or the one that issued its access token. An access-token actor has no `app` of its own, so a gate that reads `app` answers "no app" for it and falls open.
`makeActor` resolves `effectiveApp` once at construction, and `assertResolvedActor` at the request edge rejects any actor that skipped it — so reading `effectiveApp` needs no fallback. Reach for `app` only where the direct app genuinely is the question, and say why in a comment.
@@ -71,11 +67,11 @@ An [`Actor`](../src/backend/core/actor.ts) is who a request is acting as. It has
## Extensions
Extensions live alongside core ([packages/puter/extensions/](../extensions/)) and parallel the layered stack. They are meant for **non-crucial parts of the system** — things Puter still works without if removed.
Extensions live in [extensions/](../extensions/), parallel the layered stack, and hold the parts of the system Puter still works without. [AGENTS.md](../AGENTS.md#extensions) has the rule for what belongs in one.
- **Good extensions**: [thumbnails](../extensions/thumbnails.ts), [serverInfo](../extensions/serverInfo.ts), [devWatcher](../extensions/devWatcher.ts) — opt-in features cleanly bolted on.
- **Should probably be core**: [metering](../extensions/metering.ts), [appTelemetry](../extensions/appTelemetry.ts) — clients now expect these to be present, so the "extension" framing is misleading.
- **Shouldn't have been an extension**: [whoami](../extensions/whoami.ts) — it's load-bearing for every authenticated client. Keep this one in mind as a cautionary example when deciding whether something belongs in an extension.
- **Shouldn't have been an extension**: [whoami](../extensions/whoami.ts) — it's load-bearing for every authenticated client.
### Extension API
@@ -98,11 +94,11 @@ import { extension } from '@heyputer/backend/src/extensions';
const services = extension.import('service');
extension.get('/healthcheck/deep', { subdomain: 'api', adminOnly: true }, async (_req, res) => {
res.json({ ok: await services.health.runDeepCheck() });
res.json(await services.health.getStatus());
});
extension.on('user.signup', (_key, data) => {
console.log('new user', data.user.username);
extension.on('puter.signup.success', (_key, data) => {
console.log('new user', data.username);
});
```
@@ -117,10 +113,3 @@ extension.on('app.recommended', (_key, data) => {
data.appNames = ['editor', 'camera'];
});
```
## Conventions
- **TypeScript preferred** in new code where feasible. Existing JS is fine; convert opportunistically when you're already touching a file.
- **`camelCase`** for variable/function names; **`PascalCase`** for classes and for files that contain a class (`AuthService.ts`, `KVStoreDriver.ts`).
- **Deduplicate**. If two services need the same logic, lift it into a util/helper rather than calling sideways across the same layer — services should not depend on other services for code reuse.
- **Don't reach across layers.** Controllers do not poke clients directly; services do not register routes. If you find yourself wanting to, that's usually a signal the abstraction is wrong.
+20 -29
View File
@@ -12,21 +12,7 @@ Every change is backward compatible unless a maintainer has explicitly agreed to
## Core or extension?
The first decision is where the API lives.
If the API is **not crucial to core functionality — nothing in core will call it — prefer an extension** over wiring it into core. Extensions live in [extensions/](../extensions/), parallel the core layered stack, and reach core through `extension.import(...)`:
```js
const services = extension.import('service');
const stores = extension.import('store');
extension.registerDriver('myFeature', MyFeatureDriver); // first-class driver
extension.post('/my-feature/frobnicate', opts, handler); // or plain routes
```
Follow the same layered structure inside the extension (driver/controller → service → store) unless it genuinely only needs a couple of route handlers — then the lightweight `extension.get/post/...` helpers are enough on their own.
The test is the direction of dependency: Puter must still work with the extension removed. The moment core needs to call your API, it belongs in core — see [whoami](../extensions/whoami.ts) for the cautionary example of a load-bearing "extension".
The first decision is where the API lives. If **nothing in core will call it, prefer an extension** over wiring it into core. [AGENTS.md](../AGENTS.md#extensions) has the rule and its test; [architecture.md](architecture.md#extension-api) documents the `extension` API.
## Adding a new API
@@ -35,13 +21,13 @@ Work through all seven steps; the PR is complete when every one is.
### 1. Design the surface first
- Sketch the signature, options, return shape, and error cases before writing code. Find the two or three most similar existing APIs and match their conventions.
- New parameter and field names are `camelCase`. (Existing `snake_case` names stay where they already exist.)
- New parameter and field names follow the [naming rule](../AGENTS.md#language--files) (`camelCase`).
- Anything returning a list follows the [pagination convention](pagination.md): `limit`/`cursor` in, `{ items, cursor, total }` envelope out.
- Prefer an options object over a growing list of positional parameters, but keep the common case callable with a single argument where siblings do.
### 2. Backend
Follow the layered stack ([architecture.md](architecture.md)): a controller or driver at the edge, business logic in a service, persistence in a store. The edge parses and validates input and applies gates; services assume the caller is already authorized. Return exactly what the caller needs and no more — every response field you ship is permanent — and use stable `snake_case` error codes.
Follow the layered stack ([architecture.md](architecture.md), [layer rules](../AGENTS.md#backend)): a controller or driver at the edge, business logic in a service, persistence in a store. Return exactly what the caller needs and no more — every response field you ship is permanent — and use stable `snake_case` error codes.
#### Controller or driver?
@@ -54,10 +40,16 @@ Both are supported ways to define an API. **Prefer a controller when you need fi
- **Controller routes** (and extension routes — same options) take [`RouteOptions`](../src/backend/core/http/types.ts): auth gates (`requireAuth`, `requireUserActor`, `noUserSession`, `adminOnly`, `allowedAppIds`, and the access-token controls), `subdomain` routing, per-route `rateLimit`, body parsers, and arbitrary extra `middleware`. The auth flavors are subtle and default-deny — read the JSDoc on each field before picking.
- **Driver methods** get their policies from the `@Driver` options: per-method `rateLimit` (limit/window/backend), `concurrent` in-flight caps (optionally `bySubscription`), `requireSubscription`, `requireReputation`, and `noUserSession`. The `/drivers/call` surface enforces them.
- **A surface only paying accounts should reach** declares `requireSubscription` — the route option on a controller endpoint, the per-method `@Driver({ requireSubscription })` block on a driver (`/drivers/call` is one shared route, so a driver's requirement can't live in route options). `true` accepts any plan that isn't free, so a plan registered by an extension counts without core naming it; an array of policy ids (`['business', 'pro']`) accepts only those; `false` is "no requirement", the same as leaving it out. An empty array is a boot error rather than a silent no-op — it reads as subscribers-only while admitting everyone. Off unless asked for, and answered from the metering service's cached per-actor subscription, so it costs a map lookup. Distinct from `requireCredits`: this asks which plan the account is on, not whether it has budget left. Deployments with no paid plans (self-hosted installs) turn the whole thing off with `meteringEnforcement.subscriptions: false`.
- **A surface only a trusted-enough account should reach** declares `requireReputation` — the route option on a controller endpoint, the per-method `@Driver({ requireReputation })` block on a driver. The value names a tier; what that tier takes is `reputationGate.tiers` in config, so the score a surface is worth is a deployment call and can be retuned without editing the surface. A tier the running config doesn't define is inert and everyone passes — an install that doesn't score its accounts must not start turning traffic away on a score it never computed — and `reputationGate.enabled: false` stops every declared gate at once. Opt-in, implies `requireAuth`, and denies with a bare 403 `reputation_required` that names neither the score nor the tier. Nothing in the tree declares one yet: the mechanism ships ahead of the enrollment.
- **An account must have verified a specific factor** — `requireVerified` (email, and only under `strict_email_verification_required`), `requirePhoneVerified`, `requireCardVerified`. These are opt-in and require the factor to have actually been verified, unlike the default-on gate that turns away accounts still carrying a pending verification the abuse harness asked for. `requireAnyVerified: ['phone', 'card']` is the OR of the last two: any listed factor verified at any point passes, as does a paid plan where `card` is listed (a paying account has a card on file already); otherwise only the factors the deployment can verify are asked for (an SMS provider configured, a card gate an extension reports on), and with none verifiable the gate is inert rather than locking the route on a self-hosted install. It denies with the first verifiable factor's code plus `factors`, the verifiable ones in the route's order, so a client can lead with one flow and offer the other. `verifiedFactorGate.enabled: false` in config stops every declared gate at once, for an SMS-provider outage.
- **An endpoint that spends metered resources** on the caller's behalf — moving file content, making object-store requests, anything else the account is billed for — also declares `requireCredits: true`, which turns an account with nothing left of its budget away with a 402 before the handler runs. Endpoints that only describe or delete things deliberately don't: an account that has run out still has to be able to see what it has, clear it, and reach its billing pages. Drivers have no route options to declare this on, so they call `assertActorHasCredits` themselves ([src/backend/services/metering/enforcement.ts](../src/backend/services/metering/enforcement.ts)) — see `KVStoreDriver`, which does it once for every method.
- **Paying accounts only:** `requireSubscription`, as a route option on a controller endpoint or in the per-method `@Driver({ requireSubscription })` block on a driver (`/drivers/call` is one shared route, so a driver's requirement can't live in route options). `true` accepts any plan that isn't free, so a plan an extension registers counts without core naming it; an array of policy ids (`['business', 'pro']`) accepts only those; `false` or leaving it out means no requirement. An empty array is a boot error, since it would read as subscribers-only while admitting everyone. The check is a map lookup on the metering service's cached per-actor subscription. It asks which plan the account is on; `requireCredits` asks whether it has budget left. Deployments with no paid plans (self-hosted installs) turn it off with `meteringEnforcement.subscriptions: false`.
- **Trusted-enough accounts only:** `requireReputation`, as a route option or in the per-method `@Driver({ requireReputation })` block. Opt-in; implies `requireAuth`. The value names a tier, and `reputationGate.tiers` in config sets the score that tier takes, so a deployment can retune it without editing the surface. A tier the running config doesn't define is inert and everyone passes, so an install that doesn't score its accounts never turns traffic away on a score it never computed. `reputationGate.enabled: false` stops every declared gate at once. Denials are a bare 403 `reputation_required` that names neither the score nor the tier. Nothing in the tree declares one yet.
- **A specific verified factor:** `requireVerified` (email, and only under `strict_email_verification_required`), `requirePhoneVerified`, `requireCardVerified`. Opt-in, and the factor must actually have been verified, unlike the default-on gate that turns away accounts still carrying a pending verification the abuse harness asked for. `requireAnyVerified: ['phone', 'card']` is the OR of the last two: any listed factor verified at any point passes, as does a paid plan when `card` is listed (a paying account already has a card on file). Otherwise only the factors the deployment can verify are asked for (an SMS provider configured, a card gate an extension reports on); with none verifiable the gate is inert rather than locking the route on a self-hosted install. A denial carries the first verifiable factor's code plus `factors` (the verifiable ones, in the route's order), so a client can lead with one flow and offer the other. `verifiedFactorGate.enabled: false` in config stops every declared gate at once, e.g. during an SMS-provider outage.
- **Metered spend:** an endpoint that spends metered resources on the caller's behalf (moving file content, object-store requests, anything else the account is billed for) declares `requireCredits: true`, which turns an account with no budget left away with a 402 before the handler runs. Endpoints that only describe or delete things deliberately don't: an account that has run out still has to see what it has, clear it, and reach its billing pages. Drivers have no route options, so they call `assertActorHasCredits` themselves ([src/backend/services/metering/enforcement.ts](../src/backend/services/metering/enforcement.ts)) — see `KVStoreDriver`, which does it once for every method.
Provider sign-in requires the account's authenticator or a recovery code when 2FA is enabled, including signup callbacks that resolve an existing account. The five-minute challenge uses the same single-use `/login/otp` and `/login/recovery-code` verification as password sign-in. The GUI carries it in a URL fragment and removes it before initialization. A session and successful popup-return proof are issued only after verification; cancelling retains the popup's signed retry context without marking it authenticated.
Disabling 2FA through `/user-protected/disable-2fa` requires both password or OIDC revalidation and the current six-digit authenticator `code`. Missing, malformed, or incorrect codes leave the secret and recovery codes intact. A team's requirement for 2FA still prevents disabling it.
Email changes through `/user-protected/change-email` require the password or OIDC revalidation cookie, plus a six-digit `code` from the authenticator when 2FA is enabled. Missing or malformed codes return `400 bad_request`; incorrect codes return `401 code_mismatch`. Both include `factor: 'otp'`. Confirmation links are bound to that verified request. A 2FA-enabled account must request a new link if its existing link was issued without second-factor verification.
### 3. puter.js
@@ -66,33 +58,32 @@ Both are supported ways to define an API. **Prefer a controller when you need fi
### 4. Types
- Type the method where you wrote it, in JSDoc: `@param`/`@returns` on the implementation, one `@overload` block per accepted call form, and `@typedef {Object}` + `@property` for any new shape. The JSDoc is the source of truth — declarations are generated from it, so there is nothing to keep in sync by hand.
- Put a shape more than one file needs in the module's `types.js` (e.g. [src/puter-js/src/modules/kv/types.js](../src/puter-js/src/modules/kv/types.js)); anything shared across modules goes in [src/puter-js/src/lib/types.js](../src/puter-js/src/lib/types.js). A shape with one consumer can stay next to it.
- Run `npm run check:puterjs:types` — it generates the declarations and type-checks the published surface without `skipLibCheck`. **Never edit anything under `src/puter-js/types/`**: it is gitignored build output, produced by the SDK build and shipped in the npm tarball, and the next build overwrites it.
- Name the new type in [src/puter-js/index.d.ts](../src/puter-js/index.d.ts) if consumers should be able to import it. That file is the one hand-written declaration in the package: it decides what is public and re-exports nothing else.
Type the method in JSDoc on the implementation, following the [puter.js type rules](../AGENTS.md#types) (overloads, where shapes live, `index.d.ts`), then run `npm run check:puterjs:types`. Declarations are generated from the JSDoc, so there is nothing to keep in sync by hand.
### 5. Docs
- Add the method page at [src/docs/src/](../src/docs/src/)`<Area>/<method>.md` — frontmatter (`title`, `description`, `platforms`), syntax, parameters, return value, and at least one runnable example — and update the area overview (`<Area>.md`). Copy the structure of an existing page.
- The docs are the contract users code against: signatures, defaults, and return shapes match the implementation exactly.
### 6. Tests
- **Backend:** colocated Vitest tests; prefer the in-memory test server (`setupPuterTestEnv` in [src/backend/testUtil.ts](../src/backend/testUtil.ts)) over mocking.
- **SDK:** add cases to [src/puter-js/tests/api/suites/](../src/puter-js/tests/api/suites/)`<area>.suite.ts` (register new suites in `suites/index.ts`). One suite runs on node, browser, and workerd via `npm run test:puterjs` — never write per-platform tests, and rebuild first with `npm run build:workerLib` since the runners execute the built bundle.
- **Backend:** see [backend tests](../AGENTS.md#backend-tests).
- **SDK:** add cases to [src/puter-js/tests/api/suites/](../src/puter-js/tests/api/suites/)`<area>.suite.ts` (register new suites in `suites/index.ts`; there is no globbing). One suite runs on node, browser, and workerd via `npm run test:puterjs`; never write per-platform tests. The runners execute the built bundle, so run `npm run build:workerLib` first or the suite silently tests stale code.
- **Desktop-rendered UI** (`puter.ui.*`): add a Playwright spec per [src/puter-js/TESTING.md](../src/puter-js/TESTING.md).
### 7. Security pass
Scan the diff before opening the PR: no internals leaked in errors or logs, no over-broad responses, auth gates present. Flag anything auth-, permission-, or data-export-related in the PR description.
Run the [security & privacy](../AGENTS.md#security--privacy) check on the diff, and confirm the auth gates are present.
## Maintaining an existing API
Changes are **additive by default**:
- Existing call signatures keep working, including both positional and options-object forms where a method supports them. Say in the PR how existing callers are unaffected.
- New parameters are optional, with defaults that reproduce the old behavior exactly.
- Never rename, repurpose, or remove existing parameters, response fields, or error codes. Don't change types, ordering guarantees, or which fields are present when.
- New behavior that could surprise existing callers goes behind an opt-in flag.
- Docs, types, and tests move in the same PR as the behavior. A signature change with stale docs is a bug — the docs are the contract users code against.
- Docs, types, and tests move in the same PR as the behavior; a signature change with stale docs is a bug.
- Bug fixes come with a regression test that fails before the fix. Be suspicious of fixes that change observable behavior: someone may depend on the bug. When in doubt, ask a maintainer.
### Breaking changes
+11 -7
View File
@@ -28,9 +28,6 @@ Paginated responses are an envelope:
`cursor` is still returned. Never use `items.length < limit` as an
end-of-list signal.
New request params are camelCase (`includeTotal`, `fetchUntilFull`).
Pre-existing snake_case params stay for compatibility.
## Backward compatibility
Requests without pagination params keep returning the full result in the
@@ -48,10 +45,16 @@ The envelope trigger depends on the endpoint's history:
## Cursors
Cursors are opaque base64-encoded JSON, produced and consumed only by the
backend (`src/backend/util/pagination.ts`). What a cursor wraps is an
implementation detail per store:
backend (`src/backend/util/pagination.ts`). A cursor over a global sequence
(an auto-increment `id`) is sealed with `sealCursor`/`openCursor` instead, so
two pages can't be decoded into how fast the table grows; `openCursor` still
reads a plain cursor. What a cursor wraps is an implementation detail per
store:
- DynamoDB-backed lists wrap `LastEvaluatedKey`.
- DynamoDB-backed lists wrap `LastEvaluatedKey`. A kv listing of another
namespace (cross-app or through a share handle) resumes after the last entry
it returned instead, since the scan can stop on a private one, and its cursor
is sealed to that listing and padded to one length.
- SQL-backed lists wrap a keyset position — `(sortValue, id)` of the last
row — and the query seeks past it (`WHERE (col, id) > (?, ?) ORDER BY col,
id`). Any SQL list gaining a cursor must have a deterministic `ORDER BY`
@@ -91,7 +94,8 @@ hit, the response simply carries a cursor — still convention-legal.
## Adding pagination to a new endpoint
1. Use `encodeCursor`/`decodeCursor`/`normalizeLimit`/`normalizeOffset` from
`src/backend/util/pagination.ts`.
`src/backend/util/pagination.ts` — `sealCursor`/`openCursor` when the
cursor wraps a global id.
2. Push equality filters into the query so pages and counts operate on the
true result set; only genuinely per-actor filtering may remain post-query
(short-pages rule covers it).
+124 -26
View File
@@ -35,6 +35,12 @@ Optional services (compose profile `ai`, opt-in):
| `puter-ollama` | `ollama/ollama` | Local LLM provider (CPU; GPU passthrough opt-in) |
| `puter-ollama-init` | `ollama/ollama` | One-shot — pulls the default model (`tinyllama`) on first boot |
Optional services (compose profile `smtp`, opt-in):
| Container | Image | Role |
| ------------ | ------------------------ | -------------------------------------------------------- |
| `puter-smtp` | `ghcr.io/heyputer/puter` | Receives mail over SMTP and files it into user mailboxes |
State lives under `./puter/data/<service>/`.
---
@@ -144,6 +150,16 @@ Why these knobs:
- `providers.ollama.enabled: false` — Puter auto-probes a local Ollama at `127.0.0.1:11434` by default; without one running you'd see `ECONNREFUSED` on every boot. To run a bundled Ollama, see [Optional: local LLM (Ollama)](#optional-local-llm-ollama) below.
- `s3.s3Config.forcePathStyle: true` — RustFS / MinIO / fauxqs need path-style URLs (`<endpoint>/<bucket>`). Real AWS S3 wants virtual-hosted (`<bucket>.<endpoint>`) — drop this flag (or set `false`) when you swap to real S3.
- `s3.s3Config.publicEndpoint` — `endpoint` (`http://s3:9000`) only resolves inside the docker network; presigned upload/download URLs handed to the browser need a host-reachable URL. Caddy routes the `s3.<domain>` subdomain to RustFS internally and preserves the Host header end-to-end (required for S3 signature validation), so the browser hits the same port/protocol as the rest of the app — no separate published port, no mixed-content surprises when you turn on TLS. Switch to `https://s3.<your-domain>` once you enable TLS in Step 3. Real AWS S3 doesn't need this — its endpoint is already public; drop the field entirely.
- **Bucket CORS** — the browser uploads file bytes straight to RustFS with a cross-origin `PUT` to the presigned URL (Dev Center deploy, `puter.fs.upload`). That only works if the bucket answers the `OPTIONS` preflight with `Access-Control-Allow-*` headers. The `s3-init` container applies those rules on every boot (`put-bucket-cors`, origins `*` — the signed URL is the credential, and the SDK sends no cookies). If deploys fail with `No 'Access-Control-Allow-Origin' header is present`, re-run it: `docker compose run --rm s3-init`. Verify with:
```bash
curl -sk -X OPTIONS "https://s3.<domain>/<bucket>/test" \
-H "Origin: https://<domain>" \
-H "Access-Control-Request-Method: PUT" \
-H "Access-Control-Request-Headers: content-type" -D - -o /dev/null | grep -i access-control
```
You should see `access-control-allow-origin`, `allow-methods: GET, HEAD, PUT, POST, DELETE`, `allow-headers: *`, and `max-age: 3600`.
- `trust_proxy: 1` — Caddy terminates TLS and forwards `X-Forwarded-For`. Without this, `req.ip` is the docker-network address of the Caddy container instead of the real client IP, which breaks rate limiting and IP-based audit logs. `1` = one trusted hop (Caddy). Bump to `2` if you put Cloudflare in front of Caddy; never set `true` (it trusts every hop and makes XFF forgeable).
> If you ever change `MARIADB_PASSWORD` after first boot, `.env` alone won't update MariaDB — its credentials are baked into `./puter/data/mariadb/` on first init. Either rotate the password inside MariaDB by hand or `docker compose down && rm -rf ./puter/data/mariadb` to start fresh.
@@ -200,7 +216,7 @@ Drop the resulting `fullchain.pem` and `privkey.pem` into `./puter/tls/`.
```
…and update the S3 public endpoint:
```json
"s3": { "s3Config": { "publicEndpoint": "https://s3.puter.local", ... } }
"s3": { "s3Config": { "publicEndpoint": "https://s3.puter.localhost", ... } }
```
## Running behind your own reverse proxy
@@ -247,15 +263,15 @@ Healthy startup:
```
[config] override from /etc/puter/config.json
[mysql] running migrations from /opt/puter/dist/src/backend/clients/database/migrations/mysql: 2 file(s)
[mysql] running migrations from /opt/puter/dist/src/backend/clients/database/migrations/mysql: <n> file(s)
[mysql] applied mysql_mig_1.sql (...)
[mysql] applied mysql_mig_2.sql (9 statements)
...
```
Then open **<https://puter.local>** (or `http://` if you skipped TLS). Login is `admin` — the temp password is printed once in the puter container logs on first boot:
Then open **<https://puter.localhost>** (or `http://` if you skipped TLS). Login is `admin`; its temporary password is printed in the puter container logs on every boot until you change it:
```bash
docker compose logs puter | grep tmp_password
docker compose logs puter | grep "password for admin"
```
Change it in Settings after first login.
@@ -378,14 +394,14 @@ Full provider list (chat, image, video, TTS, OCR) is in the template.
### Per-user storage quota
Default is 100 MB per user.
Storage is unlimited by default (bounded by host disk). To cap each user:
```json
"storage_capacity": 5368709120, // 5 GB
"is_storage_limited": true
```
Set `is_storage_limited: false` for unlimited (bounded by host disk).
`storage_capacity` defaults to 100 MB.
### Usage metering and budgets
@@ -397,8 +413,8 @@ deletion, stay available so an account can always see what it has and clear it.
On a self-hosted install this is almost certainly not what you want. There is
nowhere to buy more, so accounts are held to the free monthly allowance
(US$0.25 of measured cost — roughly 2 GiB of downloads) and start being refused
after that. Turn it off:
(US$0.50 of measured cost for a registered account, roughly 2.5 GiB of
downloads) and start being refused after that. Turn it off:
```json
"unlimitedMetering": true
@@ -408,12 +424,6 @@ Every account then resolves to an unlimited policy. Usage is still recorded, so
the dashboard still shows what is being consumed; nothing is ever refused for
lack of budget.
For API testing, you can also set `"unlimitedMetering": true` in your
ignored runtime config: `puter/config/config.json` for Docker, or the repository's
`config.json` for `npm start`. Merge it into the existing config and restart Puter.
This includes guest accounts and applies to the whole local instance. Keep normal
metering settings when testing budget or subscription enforcement.
To keep the budgets but stop them blocking anything — recording only:
```json
@@ -454,9 +464,10 @@ temporary or permanent one.
"disable_user_signup": true
```
### Block disposable email TLDs
### Block disposable email domains
Only enforced when `env: "prod"`.
Addresses on these domains (suffix match, so subdomains too) can't sign up or be
set as an account's email.
```json
"blockedEmailDomains": ["mailinator.com", "tempmail.com", "guerrillamail.com"]
@@ -503,6 +514,95 @@ Without `--profile ai`, the `ollama` containers stay down and Puter (with `enabl
For GPU acceleration (NVIDIA), uncomment the `deploy:` block under the `ollama` service in [docker-compose.yml](../docker-compose.yml). Requires `nvidia-container-toolkit` on the host.
## Optional: receiving mail over SMTP
Puter can give each account a mailbox at a domain you control. Two pieces are
involved, and they are configured together under `userEmail`:
- an **ingress endpoint** (`POST /email/ingress`) that files a message into the
recipient's `~/.mail`. It exists whenever `userEmail.secret` is set.
- the **`puter-smtp` receiver**, which accepts mail from other mail servers and
hands each message to that endpoint. It runs as its own process, behind the
`smtp` compose profile.
> **Read this before you expose it.** An inbound mail server is unauthenticated
> by nature: anyone on the internet may hand it a message for any address in
> `localDomains`. Puter performs **no** spam filtering and **no** SPF, DKIM or
> DMARC checks — whatever is delivered is stored in the recipient's mailbox and
> counts against their storage. If that is not acceptable, put a filtering relay
> in front of it and point `localDomains` at that instead.
1. Configure it in `puter/config/config.json`:
```json
"userEmail": {
"secret": "a long random string",
"localServer": true,
"localDomains": ["example.com"]
}
```
`secret` is shared between the two halves; generate it with
`openssl rand -hex 32`. `localDomains` lists the domains this server accepts
mail for — anything else is refused at `RCPT TO`, which is what stops it
being used as an open relay. It has no default, and the receiver refuses to
start without it.
The receiver posts to `<api_base_url>/email/ingress` unless you set
`localIngressUrl`. The endpoint is served on the `api` subdomain, so if you
point it at an internal address to skip the reverse proxy, name that virtual
host too or the request will 404:
```json
"localIngressUrl": "http://puter:4100/email/ingress",
"localIngressHost": "api.example.com"
```
`localIngressHost` defaults to the host of `api_base_url`, which is usually
what you want.
2. Bring it up with the `smtp` profile:
```bash
docker compose --profile smtp up -d
docker compose logs -f puter-smtp
```
3. Point DNS at it:
```dns
mx.example.com. A 203.0.113.10
example.com. MX 10 mx.example.com.
```
Also set a `PTR` record for the address that resolves back to
`mx.example.com` — many senders refuse mail from a server without one.
Inbound TCP port 25 must reach the host. **Most cloud providers block port
25 by default and residential ISPs block it permanently**; if you cannot get
it unblocked, this will not work no matter how it is configured. The
container itself binds 2525 (it runs unprivileged) and compose publishes
`${SMTP_PORT:-25}` onto it, so you can move the host port if you need to.
A message is held in memory while it is being delivered, so peak memory is
bounded by `localMaxClients` (default 20) times the 25 MiB message ceiling.
Lower `localMaxClients` on a small host. There is no queue and no retry
schedule — if delivery fails, the sending server is told to try again later and
retrying is its job.
Mail is stored as `message/rfc822` objects under `~/.mail/objects/<date>/` and
read back through `puter.email` in the SDK.
Every setting is documented in
[config.template.jsonc](../config.template.jsonc). Two deliberate limits, both
in the name of keeping the self-hosted path simple: it does not offer STARTTLS,
so mail arrives in the clear, and it applies no per-sender rate limiting. Put a
filtering relay in front if either matters to you.
Note this is a *receiving* server only: it never sends, and it must not be used
for authenticated submission. Outbound transactional mail is the separate
`email` block above.
## Building from source instead of pulling
To run a local build against the full stack, use a source checkout and complete
@@ -518,12 +618,10 @@ services:
```
If that file already exists, merge these settings into its `puter` service.
Compose loads and merges the override automatically. `build.context: .` selects
this checkout, and `pull_policy: never` prevents pulling the published Puter image.
Use this override for local build settings instead of editing `docker-compose.yml`.
The override is ignored by Git, keeping local configuration out of pull requests
and avoiding conflicts in the shared Compose file when pulling updates.
Compose merges the override automatically: `build.context: .` builds this
checkout, and `pull_policy: never` stops it pulling the published image. Git
ignores the override, so local build settings stay out of pull requests and
don't conflict with `docker-compose.yml` when you pull updates.
From the repository root, build and start the stack:
@@ -562,11 +660,11 @@ Migrations re-apply idempotently across pulls. Volumes are preserved.
**Site loads but I get a 502 / "Bad Gateway" from Caddy.**
The puter container failed to come up. `docker compose logs puter` will tell you which dependency rejected it (most often DB password mismatch between `.env` and `config.json`).
**Login screen says "admin password not set".**
First-boot temp password is logged once. Find it: `docker compose logs puter | grep "tmp_password"`. After login, change it in Settings.
**Can't find the admin password.**
It's logged on every boot until you change it: `docker compose logs puter | grep "password for admin"`. After login, change it in Settings.
**Healthcheck reports unhealthy but the site works.**
The healthcheck hits `puter.localhost:4100/test` from inside the container. If you changed `domain` or `port`, the check still uses defaults. The site itself is fine.
The compose healthcheck requests `http://localhost:4100/` from inside the container. If you changed `port` or set `allow_all_host_values: false`, the check fails while the site itself is fine.
**Nothing resolves at `puter.example.com` after DNS changes.**
DNS propagates slowly. `dig puter.example.com` and `dig api.puter.example.com` should both return your server IP. If not, give it 5–60 minutes.
+50 -2
View File
@@ -139,8 +139,10 @@ services:
start_period: 5s
s3-init:
# One-shot container that creates the `puter-local` bucket on first
# boot. Exits 0 once the bucket exists; stays exited 0 thereafter.
# One-shot container that creates the `puter-local` bucket and
# applies its CORS rules on every boot (put-bucket-cors is
# idempotent). Without CORS, browser uploads to presigned URLs
# fail the preflight check.
image: amazon/aws-cli:latest
container_name: puter-s3-init
depends_on:
@@ -163,6 +165,26 @@ services:
echo "creating bucket $$bucket"
aws --endpoint-url "$$endpoint" s3 mb "s3://$$bucket"
fi
# Browser uploads (Dev Center deploy, puter.fs.upload) PUT to
# presigned URLs cross-origin, so the bucket must answer the
# OPTIONS preflight. Origins are open because the presigned URL
# itself is the credential and the SDK sends no cookies.
cat > /tmp/cors.json <<'EOF'
{
"CORSRules": [
{
"AllowedOrigins": ["*"],
"AllowedMethods": ["GET", "HEAD", "PUT", "POST", "DELETE"],
"AllowedHeaders": ["*"],
"ExposeHeaders": ["ETag", "x-amz-request-id"],
"MaxAgeSeconds": 3600
}
]
}
EOF
aws --endpoint-url "$$endpoint" s3api put-bucket-cors \
--bucket "$$bucket" --cors-configuration file:///tmp/cors.json
echo "bucket CORS configured"
restart: "no"
# ── Optional: local LLM ───────────────────────────────────────────
@@ -264,6 +286,32 @@ services:
retries: 3
start_period: 30s
# Inbound mail. Behind the `smtp` profile, so it only starts when asked for:
# docker compose --profile smtp up -d
# Accepts SMTP from other mail servers and hands each message to the ingress
# endpoint, which files it into the recipient's mailbox. Needs `userEmail`
# configured — see doc/self-hosting.md#optional-receiving-mail-over-smtp.
puter-smtp:
profiles: ["smtp"]
image: ghcr.io/heyputer/puter:main
pull_policy: always
container_name: puter-smtp
restart: unless-stopped
command: ["node", "-r", "./dist/src/backend/telemetry.js", "./dist/src/backend/smtp/index.js"]
depends_on:
puter:
condition: service_healthy
ports:
# The container runs unprivileged and binds 2525; 25 is what an MX record
# means, and many providers block it.
- "${SMTP_PORT:-25}:2525"
environment:
PUID: 1000
PGID: 1000
volumes:
# The same config.json the backend reads; only `userEmail` applies here.
- ./puter/config:/etc/puter:z
caddy:
image: caddy:2.11-alpine
container_name: puter-caddy
+23 -51
View File
@@ -1,5 +1,4 @@
import { Context } from '@heyputer/backend/src/core';
import type { Actor } from '@heyputer/backend/src/core/actor';
import { HttpError } from '@heyputer/backend/src/core/http';
import { PuterDriver } from '@heyputer/backend/src/drivers/types';
import type {
@@ -133,51 +132,30 @@ export class AppTelemetryDriver extends PuterDriver {
const owner = await this.stores.user.getById(ownerId);
if (!owner?.uuid) throw new HttpError(404, 'App owner not found');
const actor = Context.get('actor');
if (!actor) throw new HttpError(401, 'Authentication required');
const ownsApp = await this.services.permission
.check(actor as Actor, `apps-of-user:${owner.uuid}:write`)
.catch(() => false);
// `/drivers/call` requires auth, so there is always an actor.
const actor = Context.get('actor')!;
let ownsApp = false;
try {
ownsApp = await this.services.permission.check(
actor,
`apps-of-user:${owner.uuid}:write`,
);
} catch {
// A failed check is a denial.
}
if (!ownsApp) throw new HttpError(403, 'Permission denied');
const appId = (app as { id: number }).id;
const users = (await this.clients.db.read(
`SELECT u.id, u.username, u.uuid, u.email FROM user_to_app_permissions p
INNER JOIN ${this.clients.db.quoteIdentifier('user')} u ON p.user_id = u.id
WHERE p.permission = 'flag:app-is-authenticated' AND p.app_id = ?
ORDER BY (p.dt IS NOT NULL), p.dt, p.user_id
LIMIT ? OFFSET ?`,
[appId, safeLimit, safeOffset],
)) as Array<{
id: number;
username: string;
uuid: string;
email: string | null;
}>;
// Only surface a user's email if *that user* granted this app the
// `user:<their-uuid>:email:read` permission — the same grant
// `puter.perms.requestEmail()` obtains and `whoami` honours. This is a
// per-user check keyed on the app (not the calling owner-actor): a
// user may have authenticated into the app without sharing their
// email. Resolve the whole page in one query.
const emailPermitted = new Set<number>();
if (users.length > 0) {
const permStrings = users.map((u) => `user:${u.uuid}:email:read`);
const placeholders = permStrings.map(() => '?').join(', ');
const grants = (await this.clients.db.read(
`SELECT user_id FROM user_to_app_permissions
WHERE app_id = ? AND permission IN (${placeholders})`,
[appId, ...permStrings],
)) as Array<{ user_id: number }>;
for (const g of grants) emailPermitted.add(g.user_id);
}
return users.map((e) =>
emailPermitted.has(e.id)
? { user: e.username, user_uuid: e.uuid, user_email: e.email }
: { user: e.username, user_uuid: e.uuid },
// An email surfaces only for a user who granted *this app*
// `user:<their-uuid>:email:read` — the grant `puter.perms.requestEmail()`
// obtains and `whoami` honours. Authenticating alone doesn't share it.
const users = await this.stores.permission.listAppAuthenticatedUsers(
(app as { id: number }).id,
{ limit: safeLimit, offset: safeOffset },
);
return users.map((u) =>
u.emailShared
? { user: u.username, user_uuid: u.uuid, user_email: u.email }
: { user: u.username, user_uuid: u.uuid },
);
}
@@ -190,13 +168,7 @@ export class AppTelemetryDriver extends PuterDriver {
const app = await this.stores.app.getByUid(app_uuid);
if (!app) throw new HttpError(404, 'App not found');
const [row] = (await this.clients.db.read(
`SELECT COUNT(*) AS n FROM user_to_app_permissions
WHERE permission = 'flag:app-is-authenticated' AND app_id = ?`,
[(app as { id: number }).id],
)) as Array<{ n: number }>;
return row?.n ?? 0;
return this.stores.permission.countAppUsers((app as { id: number }).id);
}
}
-9
View File
@@ -74,15 +74,6 @@ const grantInstalled = async (appId: number, userId: number) => {
};
describe('installedApps extension — handleInstalledApps', () => {
it('throws HttpError(401) when no actor is on the context', async () => {
const { res } = makeRes();
await expect(
runWithContext({ actor: undefined }, () =>
handleInstalledApps(makeReq({}), res),
),
).rejects.toMatchObject({ statusCode: 401 });
});
it('throws HttpError(400) when orderBy is not in the allowlist', async () => {
const user = await seedUser();
const { res } = makeRes();
+16 -43
View File
@@ -2,68 +2,41 @@ import type { Request, Response } from 'express';
import { Context } from '@heyputer/backend/src/core';
import { HttpError } from '@heyputer/backend/src/core/http';
import { extension } from '@heyputer/backend/src/extensions';
import {
INSTALLED_APPS_ORDERS,
type InstalledAppsOrder,
} from '@heyputer/backend/src/stores/permission/PermissionStore.js';
import {
getAppIconCdnUrl,
getAppIconUrl,
} from '@heyputer/backend/src/util/appIcon.js';
const clients = extension.import('client');
const ALLOWED_ORDER_BY = [
'id',
'name',
'uid',
'title',
'installed_at',
] as const;
const ORDER_BY_FIELD_MAP: Record<string, string> = {
id: 'apps.id',
name: 'apps.name',
uid: 'apps.uid',
title: 'apps.title',
installed_at: 'installed_at',
};
const stores = extension.import('store');
export const handleInstalledApps = async (
req: Request,
res: Response,
): Promise<void> => {
const actor = Context.get('actor');
if (!actor?.user?.id) throw new HttpError(401, 'Authentication required');
// Behind `requireUserActor`, so there is always an actor.
const actor = Context.get('actor')!;
const orderBy = String(req.query.orderBy ?? 'installed_at');
if (!(ALLOWED_ORDER_BY as readonly string[]).includes(orderBy)) {
const orderBy = String(
req.query.orderBy ?? 'installed_at',
) as InstalledAppsOrder;
if (!INSTALLED_APPS_ORDERS.includes(orderBy)) {
throw new HttpError(
400,
`Invalid orderBy. Allowed: ${ALLOWED_ORDER_BY.join(', ')}`,
`Invalid orderBy. Allowed: ${INSTALLED_APPS_ORDERS.join(', ')}`,
);
}
const page = Math.max(Number(req.query.page) || 1, 1);
const limit = Math.min(Math.max(Number(req.query.limit) || 100, 1), 100);
const offset = (page - 1) * limit;
const orderByField = ORDER_BY_FIELD_MAP[orderBy];
const sortDirection = req.query.desc ? 'DESC' : 'ASC';
const installedApps = (await clients.db.read(
`SELECT
apps.name,
apps.uid,
apps.title,
apps.description,
apps.icon,
apps.index_url,
apps.owner_user_id,
MIN(perm.dt) AS installed_at
FROM apps
LEFT JOIN user_to_app_permissions AS perm ON apps.id = perm.app_id
WHERE perm.user_id = ?
GROUP BY apps.id, apps.name, apps.uid, apps.title, apps.description
ORDER BY ${orderByField} ${sortDirection}
LIMIT ?
OFFSET ?`,
[actor.user.id, limit, offset],
)) as Array<Record<string, unknown>>;
const installedApps = await stores.permission.listAppsGrantedByUser(
actor.user.id!,
{ orderBy, descending: Boolean(req.query.desc), limit, offset },
);
const apiBaseUrl = extension.config.api_base_url as string | undefined;
res.json(
+202 -21
View File
@@ -8,6 +8,7 @@ import {
it,
vi,
} from 'vitest';
import { makeActor, type Actor } from '../src/backend/core/actor.ts';
import { runWithContext } from '../src/backend/core/context.ts';
import { PuterServer } from '../src/backend/server.ts';
import { setupTestServer } from '../src/backend/testUtil.ts';
@@ -62,15 +63,6 @@ const seedUser = async () => {
};
describe('metering extension — handleMeteringUsage', () => {
it('throws HttpError(401) when no user actor is on the context', async () => {
const { res } = makeRes();
await expect(
runWithContext({ actor: undefined }, () =>
handleMeteringUsage(makeReq(), res),
),
).rejects.toMatchObject({ statusCode: 401 });
});
it('returns usage details merged with allowanceInfo for an authenticated user', async () => {
const user = await seedUser();
const { res, captured } = makeRes();
@@ -88,21 +80,83 @@ describe('metering extension — handleMeteringUsage', () => {
(captured.body as Record<string, unknown>).allowanceInfo,
).toBeDefined();
});
describe('usage split across apps', () => {
const seedUsage = async () => {
const user = await seedUser();
const owner = { uuid: user.uuid, id: user.id as number };
const metering = server.services.metering;
await metering.incrementUsage(
makeActor({ user: owner, app: { uid: 'app-mine' } }),
'kv:read',
1,
10,
);
await metering.incrementUsage(
makeActor({ user: owner, app: { uid: 'app-other' } }),
'kv:write',
1,
20,
);
await metering.incrementUsage(
makeActor({ user: owner }),
'kv:list',
1,
40,
);
await server.stores.meteringBuffer.flushCycle();
return owner;
};
const usageAs = async (actor: Actor) => {
const { res, captured } = makeRes();
await runWithContext({ actor }, () =>
handleMeteringUsage(makeReq(), res),
);
return captured.body as {
usage: Record<string, unknown> & { total: number };
appTotals: Record<string, unknown>;
allowanceInfo: Record<string, unknown>;
};
};
it('reports only the calling app’s own usage to an app actor', async () => {
const owner = await seedUsage();
const body = await usageAs(
makeActor({ user: owner, app: { uid: 'app-mine' } }),
);
expect(body.usage.total).toBe(10);
expect(Object.keys(body.usage).sort()).toEqual([
'kv:read',
'total',
]);
expect(body.appTotals).toEqual({
'app-mine': { total: 10, count: 1 },
});
expect(body.allowanceInfo).toMatchObject({
remaining: expect.any(Number),
monthUsageAllowance: expect.any(Number),
});
});
it('still reports the whole account to the user', async () => {
const owner = await seedUsage();
const body = await usageAs(makeActor({ user: owner }));
expect(body.usage.total).toBe(70);
expect(body.usage).toHaveProperty('kv:read');
expect(body.usage).toHaveProperty('kv:write');
expect(body.usage).toHaveProperty('kv:list');
expect(body.appTotals).toMatchObject({
'app-mine': { total: 10, count: 1 },
'app-other': { total: 20, count: 1 },
});
});
});
});
describe('metering extension — handleMeteringUsageForApp', () => {
it('throws HttpError(401) when no user actor is on the context', async () => {
const { res } = makeRes();
await expect(
runWithContext({ actor: undefined }, () =>
handleMeteringUsageForApp(
makeReq({ params: { appIdOrName: 'any' } }),
res,
),
),
).rejects.toMatchObject({ statusCode: 401 });
});
it('throws HttpError(400) when no appId is supplied', async () => {
const user = await seedUser();
const { res } = makeRes();
@@ -132,6 +186,96 @@ describe('metering extension — handleMeteringUsageForApp', () => {
),
).rejects.toMatchObject({ statusCode: 404 });
});
describe('scoped to the calling app', () => {
const appUsageAs = async (actor: Actor, appIdOrName: string) => {
const { res, captured } = makeRes();
await runWithContext({ actor }, () =>
handleMeteringUsageForApp(
makeReq({ params: { appIdOrName } }),
res,
),
);
return captured.body as { total: number };
};
const seedApps = async () => {
const user = await seedUser();
const owner = { uuid: user.uuid, id: user.id as number };
const mine = `app-${uuidv4()}`;
const other = `app-${uuidv4()}`;
await server.services.metering.incrementUsage(
makeActor({ user: owner, app: { uid: mine } }),
'kv:read',
1,
10,
);
await server.services.metering.incrementUsage(
makeActor({ user: owner, app: { uid: other } }),
'kv:write',
1,
20,
);
await server.stores.meteringBuffer.flushCycle();
return {
owner,
mine,
other,
asMine: makeActor({ user: owner, app: { uid: mine } }),
};
};
it('lets an app read its own usage', async () => {
const { mine, asMine } = await seedApps();
expect((await appUsageAs(asMine, mine)).total).toBe(10);
});
it('refuses an app another app’s usage', async () => {
const { other, asMine } = await seedApps();
await expect(appUsageAs(asMine, other)).rejects.toMatchObject({
statusCode: 403,
});
});
it('refuses an app another app’s usage by name', async () => {
const { owner, other, asMine } = await seedApps();
const name = `other-${uuidv4()}`;
await server.clients.db.write(
'INSERT INTO `apps` (`uid`, `name`, `title`, `index_url`, `owner_user_id`) VALUES (?, ?, ?, ?, ?)',
[other, name, name, `https://${name}.example/`, owner.id],
);
await expect(appUsageAs(asMine, name)).rejects.toMatchObject({
statusCode: 403,
});
});
it('refuses an app the usage outside any app', async () => {
const { asMine } = await seedApps();
await expect(appUsageAs(asMine, 'os-global')).rejects.toMatchObject(
{ statusCode: 403 },
);
});
it('still lets the user read any of their apps', async () => {
const { owner, mine, other } = await seedApps();
const asUser = makeActor({ user: owner });
expect((await appUsageAs(asUser, mine)).total).toBe(10);
expect((await appUsageAs(asUser, other)).total).toBe(20);
});
it('lets the user read their usage outside any app', async () => {
const { owner } = await seedApps();
const asUser = makeActor({ user: owner });
await server.services.metering.incrementUsage(
asUser,
'kv:read',
1,
5,
);
await server.stores.meteringBuffer.flushCycle();
expect((await appUsageAs(asUser, 'os-global')).total).toBe(5);
});
});
});
describe('metering extension — handleMeteringGlobalUsage', () => {
@@ -165,4 +309,41 @@ describe('metering extension — handleMeteringAllCosts', () => {
// per request).
expect(costsA).toBe(costsB);
});
it('refreshes a warm catalogue at the UTC price boundary', async () => {
vi.useFakeTimers({ toFake: ['Date'] });
try {
vi.setSystemTime(new Date('2026-11-30T23:59:59Z'));
const before = makeRes();
await handleMeteringAllCosts(makeReq(), before.res);
vi.setSystemTime(new Date('2026-12-01T00:00:00Z'));
const after = makeRes();
await handleMeteringAllCosts(makeReq(), after.res);
expect(
(after.captured.body as { costs: unknown[] }).costs,
).not.toBe((before.captured.body as { costs: unknown[] }).costs);
} finally {
vi.useRealTimers();
}
});
// A scheduled price change has to reach the catalogue without a restart.
it('collects the catalogue again once the cache expires', async () => {
vi.useFakeTimers({ toFake: ['Date'] });
try {
vi.setSystemTime(new Date('2026-12-02T00:00:00Z'));
const a = makeRes();
const b = makeRes();
await handleMeteringAllCosts(makeReq(), a.res);
vi.setSystemTime(Date.now() + 10 * 60 * 1000 + 1);
await handleMeteringAllCosts(makeReq(), b.res);
const costsA = (a.captured.body as { costs: unknown[] }).costs;
const costsB = (b.captured.body as { costs: unknown[] }).costs;
expect(costsB).not.toBe(costsA);
expect(costsB).toEqual(costsA);
} finally {
vi.useRealTimers();
}
});
});
+20 -20
View File
@@ -6,6 +6,7 @@ import {
servicesContainers,
} from '@heyputer/backend/src/exports';
import { extension } from '@heyputer/backend/src/extensions';
import { GLOBAL_APP_KEY } from '@heyputer/backend/src/services/metering/consts';
import {
creditMultiplierFrom,
toCredits,
@@ -13,11 +14,13 @@ import {
import type { Request, Response } from 'express';
const services = extension.import('service');
const clients = extension.import('client');
const stores = extension.import('store');
// Cached on first request — the underlying cost catalogues are baked into
// driver/controller source so they only change on deploy.
// Catalogues change on deploy or on a scheduled price change, so the cache
// expires rather than living as long as the process.
const ALL_COSTS_TTL_MS = 10 * 60 * 1000;
let cachedAllCosts: Record<string, unknown>[] | null = null;
let allCostsExpireAt = 0;
function collectAllCosts(): Record<string, unknown>[] {
const all: Record<string, unknown>[] = [];
@@ -94,8 +97,8 @@ export const handleMeteringUsage = async (
_req: Request,
res: Response,
): Promise<void> => {
const actor = Context.get('actor');
if (!actor?.user) throw new HttpError(401, 'Authentication required');
// Behind `requireAuth`, so there is always an actor.
const actor = Context.get('actor')!;
const [actorUsage, allowanceInfo] = await Promise.all([
services.metering.getActorCurrentMonthUsageDetails(actor),
@@ -145,23 +148,18 @@ export const handleMeteringUsageForApp = async (
req: Request,
res: Response,
): Promise<void> => {
const actor = Context.get('actor');
if (!actor?.user) throw new HttpError(401, 'Authentication required');
// Behind `requireAuth`, so there is always an actor.
const actor = Context.get('actor')!;
let appId = String(req.params.appIdOrName ?? '');
if (!appId) throw new HttpError(400, 'appId parameter is required');
// If not a UUID-shaped app UID, look up by name
if (!appId.startsWith('app-')) {
const appRows = (await clients.db.read(
'SELECT `uid` FROM `apps` WHERE `name` = ? LIMIT 1',
[appId],
)) as Array<{ uid: string }>;
if (appRows.length > 0) {
appId = appRows[0].uid;
} else {
throw new HttpError(404, 'App not found');
}
// If not a UUID-shaped app UID or the global sentinel, look up by name.
// Which apps an actor may read is MeteringService's call, not this route's.
if (!appId.startsWith('app-') && appId !== GLOBAL_APP_KEY) {
const app = await stores.app.getByName(appId);
if (!app) throw new HttpError(404, 'App not found');
appId = app.uid;
}
const appUsage =
@@ -191,12 +189,14 @@ export const handleMeteringGlobalUsage = async (
res.json(globalUsage);
};
// First hit walks the registries; subsequent hits serve the in-memory cache.
// A hit walks the registries when the in-memory cache is missing or expired.
export const handleMeteringAllCosts = async (
_req: Request,
res: Response,
): Promise<void> => {
if (!cachedAllCosts) {
if (!cachedAllCosts || Date.now() >= allCostsExpireAt) {
allCostsExpireAt =
(Math.floor(Date.now() / ALL_COSTS_TTL_MS) + 1) * ALL_COSTS_TTL_MS;
const raw = collectAllCosts();
const multiplier = creditMultiplierFrom(extension.config);
// Price the catalogue in the same unit the usage endpoints report:
File diff suppressed because it is too large. Load diff
+206 -86
View File
@@ -1,15 +1,21 @@
import {
CopyObjectCommand,
DeleteObjectCommand,
GetObjectCommand,
HeadObjectCommand,
PutObjectCommand,
S3Client,
} from '@aws-sdk/client-s3';
import { getSignedUrl } from '@aws-sdk/s3-request-presigner';
import { extension } from '@heyputer/backend/src/extensions';
import type { FSEntryStore } from '@heyputer/backend/src/stores/fs/FSEntryStore';
import { isMissingObjectError } from '@heyputer/backend/src/stores/fs/S3ObjectStore';
import crypto from 'node:crypto';
import type { Readable } from 'node:stream';
import sharp from 'sharp';
const clients = extension.import('client');
const stores = extension.import('store');
type ThumbnailEntryStore = Pick<FSEntryStore, 'updateEntryThumbnailByUuid'>;
const MAX_THUMBNAIL_BYTES = 2 * 1024 * 1024;
const MAX_THUMBNAIL_PIXELS = 64e6;
@@ -22,8 +28,13 @@ const THUMBNAIL_KEY_PREFIX = 'thumbnails/';
const UUID_PATTERN =
/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;
const mintThumbnailKey = (): string =>
`${THUMBNAIL_KEY_PREFIX}${crypto.randomUUID()}`;
const isUuid = (value: unknown): value is string =>
typeof value === 'string' && UUID_PATTERN.test(value);
// `thumbnails/<entry uuid>/<random uuid>`: the entry segment binds the object
// to the entry it was minted for, which is what writes and deletes check.
const mintThumbnailKey = (entryUuid: string): string =>
`${THUMBNAIL_KEY_PREFIX}${entryUuid}/${crypto.randomUUID()}`;
/**
* Extract the object key from a stored thumbnail pointer, or null when the
@@ -31,12 +42,13 @@ const mintThumbnailKey = (): string =>
*
* `fsentries.thumbnail` is writable through the FS API, so neither half of the
* stored string is trusted: the bucket is discarded (callers always pass their
* own) and the key must sit under {@link THUMBNAIL_KEY_PREFIX} with a random
* uuid. Honouring an arbitrary key would lend this extension's storage
* credentials to whatever object the caller named — in the shared-bucket layout
* that is every user's file, since an fs object's key is its fsentry uuid.
* Legacy bare-uuid thumbnails fail the check and are treated as absent; they
* are indistinguishable from a planted pointer, so there is nothing safer to do
* own) and the key must sit under {@link THUMBNAIL_KEY_PREFIX} as either
* `<uuid>` (minted before keys were bound to entries) or `<entry uuid>/<uuid>`.
* Honouring an arbitrary key would lend this extension's storage credentials to
* whatever object the caller named — in the shared-bucket layout that is every
* user's file, since an fs object's key is its fsentry uuid. Bare-uuid keys
* from before the prefix fail the check and are treated as absent; they are
* indistinguishable from a planted pointer, so there is nothing safer to do
* with them than stop signing them.
*/
const resolveThumbnailKey = (pointer: string): string | null => {
@@ -58,10 +70,26 @@ const resolveThumbnailKey = (pointer: string): string | null => {
key = segments.join('/');
}
if (!key.startsWith(THUMBNAIL_KEY_PREFIX)) return null;
const id = key.slice(THUMBNAIL_KEY_PREFIX.length);
return UUID_PATTERN.test(id) ? key : null;
const segments = key.slice(THUMBNAIL_KEY_PREFIX.length).split('/');
return segments.length <= 2 && segments.every(isUuid) ? key : null;
};
// The entry a resolved key was minted for. Null for an unbound key, whose
// holder can't be told apart from a row that copied or planted the pointer.
const thumbnailKeyOwner = (key: string): string | null => {
const segments = key.slice(THUMBNAIL_KEY_PREFIX.length).split('/');
return segments.length === 2 ? segments[0] : null;
};
// Whether a stored thumbnail names an object in our storage. Anything else is
// an external image URL the client supplied: shown as given, never copied or
// deleted.
const isStoragePointer = (pointer: string, bucketEndpoint: string): boolean =>
pointer.startsWith('s3://') ||
// Legacy format — remove after full migration
(pointer.startsWith('https') &&
pointer.includes(new URL(bucketEndpoint).hostname));
// S3 client + bucket config — lazily resolved after boot from config.
let s3Client: S3Client | null = null;
let s3PresignClient: S3Client | null = null;
@@ -111,13 +139,6 @@ function getPresignClient(): S3Client {
return resolveClients().presign;
}
function base64ParseDataUrl(dataURL: string) {
dataURL = dataURL.slice(5);
const mimeType = dataURL.split(';')[0];
const data = Buffer.from(dataURL.split(',')[1], 'base64');
return { mimeType, data };
}
// Strictly decode a data: URL and validate the decoded image. Encoded-string
// length lies about decoded byte count (whitespace, padding) and says nothing
// about pixel count — a 2MB PNG can decompress to hundreds of MB of raster.
@@ -144,16 +165,73 @@ async function decodeAndValidateThumbnail(
return { mimeType, data };
}
// A storage pointer a client hands back after a presigned upload is kept only
// if it names a key minted for this entry and an object within the size bound.
async function vetUploadedThumbnail(
pointer: string,
entryUuid: unknown,
deps: { s3: S3Client; bucketName: string },
): Promise<string | null> {
const key = resolveThumbnailKey(pointer);
if (!key || thumbnailKeyOwner(key) !== entryUuid) return null;
let size: number | undefined;
try {
const head = await deps.s3.send(
new HeadObjectCommand({ Bucket: deps.bucketName, Key: key }),
);
size = head.ContentLength;
} catch (err) {
if (!isMissingObjectError(err)) {
console.warn(
'[thumbnails] failed to check uploaded thumbnail',
err,
);
}
return null;
}
if (typeof size !== 'number') return null;
if (size > MAX_THUMBNAIL_BYTES) {
// Only reachable on a store that doesn't enforce the signed length.
// The key is this entry's own, so removing it is safe.
try {
await deps.s3.send(
new DeleteObjectCommand({ Bucket: deps.bucketName, Key: key }),
);
} catch (err) {
console.warn(
'[thumbnails] failed to remove oversized thumbnail',
err,
);
}
return null;
}
return `s3://${deps.bucketName}/${key}`;
}
// -- thumbnail.created -----------------------------------------------
// Intercept data-URL thumbnails before they hit the DB: upload to S3
// and replace the URL with an s3:// pointer.
// and replace the URL with an s3:// pointer bound to the entry.
export async function handleThumbnailCreated(
event: Record<string, unknown>,
deps: { s3: S3Client; bucketName: string },
deps: { s3: S3Client; bucketName: string; bucketEndpoint: string },
): Promise<void> {
const url = event.url;
if (typeof url !== 'string' || !url.startsWith('data:')) return;
if (typeof url !== 'string') return;
if (!url.startsWith('data:')) {
if (isStoragePointer(url, deps.bucketEndpoint)) {
event.url = await vetUploadedThumbnail(url, event.uuid, deps);
}
return;
}
const entryUuid = event.uuid;
if (!isUuid(entryUuid)) {
event.url = null;
return;
}
const decoded = await decodeAndValidateThumbnail(url);
if (!decoded) {
@@ -161,7 +239,7 @@ export async function handleThumbnailCreated(
return;
}
const key = mintThumbnailKey();
const key = mintThumbnailKey(entryUuid);
event.url = `s3://${deps.bucketName}/${key}`;
await deps.s3.send(
@@ -190,21 +268,24 @@ export const handleThumbnailUploadPrepare = async (
typeof item.contentType === 'string' ? item.contentType.trim() : '';
if (!contentType) continue;
if (item.size !== undefined) {
const size = Number(item.size);
if (
!Number.isFinite(size) ||
size < 0 ||
size > MAX_THUMBNAIL_BYTES
)
continue;
}
const size = item.size;
if (
typeof size !== 'number' ||
!Number.isInteger(size) ||
size < 0 ||
size > MAX_THUMBNAIL_BYTES
)
continue;
if (!isUuid(item.item_uid)) continue;
const key = mintThumbnailKey();
const key = mintThumbnailKey(item.item_uid);
const command = new PutObjectCommand({
Bucket: deps.bucketName,
Key: key,
ContentType: contentType,
// Signing covers `content-length`, so a body of any other size
// fails the signature instead of landing.
ContentLength: size,
});
item.uploadUrl = await getSignedUrl(presignClient, command, {
expiresIn: 900,
@@ -220,19 +301,14 @@ export const handleThumbnailRead = async (
s3Presign: S3Client;
bucketName: string;
bucketEndpoint: string;
db: { write: (sql: string, params: unknown[]) => Promise<unknown> };
fsEntry: ThumbnailEntryStore;
},
): Promise<void> => {
const thumb = entry.thumbnail;
if (typeof thumb !== 'string' || !thumb) return;
const presignClient = deps.s3Presign;
if (
thumb.startsWith('s3://') ||
// Legacy format — remove after full migration
(thumb.startsWith('https') &&
thumb.includes(new URL(deps.bucketEndpoint).hostname))
) {
if (isStoragePointer(thumb, deps.bucketEndpoint)) {
const key = resolveThumbnailKey(thumb);
if (!key) {
// Not a pointer we minted — refuse to sign it rather than hand
@@ -246,31 +322,52 @@ export const handleThumbnailRead = async (
{ expiresIn: 604800 },
);
} else if (thumb.startsWith('data')) {
// Inline data-URL migration: upload to S3 and update the DB entry.
const key = mintThumbnailKey();
const { mimeType, data } = base64ParseDataUrl(thumb);
const newUrl = `s3://${deps.bucketName}/${key}`;
// Move an inline thumbnail into storage. The row only takes the new
// pointer while it still holds this data URL, so of concurrent reads
// one wins and the rest discard their upload.
const uuid = entry.uuid ?? entry.uid;
if (!isUuid(uuid)) return;
// Same bar as an upload; anything else stays inline, unmigrated.
const decoded = await decodeAndValidateThumbnail(thumb);
if (!decoded) return;
const key = mintThumbnailKey(uuid);
await deps.s3.send(
new PutObjectCommand({
Bucket: deps.bucketName,
Key: key,
Body: data,
ContentType: mimeType,
Body: decoded.data,
ContentType: decoded.mimeType,
}),
);
// Best-effort async DB update
const uuid = entry.uuid ?? entry.uid;
if (uuid) {
deps.db
.write(
'UPDATE `fsentries` SET `thumbnail` = ? WHERE `uuid` = ?',
[newUrl, uuid],
)
.catch((err: unknown) =>
console.warn('[thumbnails] inline migration failed', err),
let migrated: boolean;
try {
migrated = await deps.fsEntry.updateEntryThumbnailByUuid(
uuid,
`s3://${deps.bucketName}/${key}`,
thumb,
);
} catch (err) {
// The row may or may not point at the object now, so keep it.
console.warn('[thumbnails] inline migration failed', err);
return;
}
if (!migrated) {
try {
await deps.s3.send(
new DeleteObjectCommand({
Bucket: deps.bucketName,
Key: key,
}),
);
} catch (err) {
console.warn(
'[thumbnails] failed to remove unused thumbnail',
err,
);
}
return;
}
entry.thumbnail = await getSignedUrl(
@@ -286,63 +383,83 @@ export const handleFsCopyNodeThumbnail = async (
deps: {
s3: S3Client;
bucketName: string;
db: { write: (sql: string, params: unknown[]) => Promise<unknown> };
bucketEndpoint: string;
fsEntry: ThumbnailEntryStore;
},
): Promise<void> => {
const copy = payload.copy;
const thumbnailUrl = copy?.thumbnail;
if (!copy || !copy.uuid || typeof thumbnailUrl !== 'string') return;
if (!copy || !isUuid(copy.uuid) || typeof thumbnailUrl !== 'string') return;
if (!isStoragePointer(thumbnailUrl, deps.bucketEndpoint)) return;
// Same trust rule as the read and remove paths: only touch objects this
// extension minted.
const sourceKey = resolveThumbnailKey(thumbnailUrl);
if (!sourceKey) return;
// The copied row points at the SAME S3 object as its source, and
// fs.remove.node deletes the pointed-to object — so the first removal
// among the sharers (an overwrite, a trash purge) would break every
// other sharer's thumbnail. Give the copy an object of its own.
const newKey = mintThumbnailKey();
// The copied row points at the SAME object as its source. Give the copy
// an object of its own, bound to it, so removing either entry leaves the
// other's thumbnail alone.
const newKey = mintThumbnailKey(copy.uuid);
try {
// Nothing larger than an upload may produce gets duplicated; such an
// object came in through an unbounded upload URL.
const head = await deps.s3.send(
new HeadObjectCommand({ Bucket: deps.bucketName, Key: sourceKey }),
);
if (
typeof head.ContentLength !== 'number' ||
head.ContentLength > MAX_THUMBNAIL_BYTES
) {
throw new Error('thumbnail exceeds the size bound');
}
// Read and rewrite rather than CopyObject: a copy source skips any path
// in the client's endpoint, so it can miss an object Bucket/Key reach.
const source = await deps.s3.send(
new GetObjectCommand({ Bucket: deps.bucketName, Key: sourceKey }),
);
// The object can be replaced after the HEAD; bound what gets buffered.
if ((source.ContentLength ?? 0) > MAX_THUMBNAIL_BYTES) {
(source.Body as Readable | undefined)?.destroy();
throw new Error('thumbnail exceeds the size bound');
}
const body = await source.Body?.transformToByteArray();
if (!body) throw new Error('thumbnail has no body');
await deps.s3.send(
new CopyObjectCommand({
new PutObjectCommand({
Bucket: deps.bucketName,
CopySource: `${deps.bucketName}/${sourceKey}`,
Key: newKey,
Body: body,
ContentType: source.ContentType,
}),
);
} catch (err) {
// The shared object is already gone (e.g. a sharer was removed
// before this fix existed) — the pointer is dead either way, so
// drop it rather than leave the row advertising a thumbnail it
// doesn't have.
await deps.db.write(
'UPDATE `fsentries` SET `thumbnail` = NULL WHERE `uuid` = ?',
[copy.uuid],
);
// The shared object is gone or oversized — drop the pointer rather
// than leave the row advertising a thumbnail it doesn't have.
await deps.fsEntry.updateEntryThumbnailByUuid(copy.uuid, null);
console.warn('[thumbnails] failed to duplicate thumbnail on copy', err);
return;
}
await deps.db.write(
'UPDATE `fsentries` SET `thumbnail` = ? WHERE `uuid` = ?',
[`s3://${deps.bucketName}/${newKey}`, copy.uuid],
await deps.fsEntry.updateEntryThumbnailByUuid(
copy.uuid,
`s3://${deps.bucketName}/${newKey}`,
);
};
export const handleFsRemoveNodeThumbnail = async (
payload: { target: { thumbnail?: string | null } },
deps: { s3: S3Client; bucketName: string },
payload: { target: { thumbnail?: string | null; uuid?: string } },
deps: { s3: S3Client; bucketName: string; bucketEndpoint: string },
): Promise<void> => {
const thumbnailUrl = payload.target.thumbnail;
if (!thumbnailUrl) return;
const { thumbnail: thumbnailUrl, uuid } = payload.target;
if (!thumbnailUrl || !isStoragePointer(thumbnailUrl, deps.bucketEndpoint))
return;
// Same trust rule as the read path, and load-bearing for the same reason:
// the pointer decides which object gets deleted, so a key we didn't mint
// would let the owner of one file destroy an object belonging to someone
// else just by naming it here.
// The pointer decides which object gets deleted, and a key is visible to
// anyone who can list the entry holding it. Only a key minted for this
// entry is deleted; an unbound one may be shared or planted, so it stays.
const key = resolveThumbnailKey(thumbnailUrl);
if (!key) return;
if (!key || thumbnailKeyOwner(key) !== uuid) return;
await deps.s3.send(
new DeleteObjectCommand({ Bucket: deps.bucketName, Key: key }),
@@ -355,6 +472,7 @@ extension.on(
await handleThumbnailCreated(event, {
s3: getClient(),
bucketName: thumbnailBucketName,
bucketEndpoint: extensionBucketEndpoint,
});
},
);
@@ -381,7 +499,7 @@ extension.on('thumbnail.read', async (_key, entry: Record<string, unknown>) => {
s3Presign: getPresignClient(),
bucketName: thumbnailBucketName,
bucketEndpoint: extensionBucketEndpoint,
db: clients.db,
fsEntry: stores.fsEntry,
});
});
@@ -397,7 +515,8 @@ extension.on('fs.copy.node', async (_key, payload) => {
{
s3: getClient(),
bucketName: thumbnailBucketName,
db: clients.db,
bucketEndpoint: extensionBucketEndpoint,
fsEntry: stores.fsEntry,
},
);
});
@@ -409,5 +528,6 @@ extension.on('fs.remove.node', async (_key, payload) => {
await handleFsRemoveNodeThumbnail(payload, {
s3: getClient(),
bucketName: thumbnailBucketName,
bucketEndpoint: extensionBucketEndpoint,
});
});
+355
View File
@@ -0,0 +1,355 @@
import { EventEmitter } from 'node:events';
import { Readable } from 'node:stream';
import type { Request, Response } from 'express';
import { beforeEach, describe, expect, test, vi } from 'vitest';
const SECRET = 'ingress-secret';
// Only `extension` is stubbed; the secret comparison and the routing are the
// real code paths, and the fake service/store layer is the boundary.
const state = vi.hoisted(() => ({
config: { userEmail: { secret: 'ingress-secret' } } as Record<
string,
unknown
>,
routes: [] as Array<{
path: string;
handler: (req: unknown, res: unknown) => unknown;
}>,
users: new Map<string, Record<string, unknown>>(),
writes: [] as Array<{
userId: number;
path: string;
homeRegion?: string;
content?: unknown;
}>,
feedback: [] as Array<{ to: string; from: string | null; raw: Buffer }>,
takeFeedback: true,
}));
vi.mock('@heyputer/backend/src/extensions', () => ({
extension: {
get config() {
return state.config;
},
post(
path: string,
_opts: unknown,
handler: (req: unknown, res: unknown) => unknown,
) {
state.routes.push({ path, handler });
},
import(layer: string) {
if (layer === 'client') {
return {
event: {
async emitAndWait(
key: string,
data: {
to: string;
from: string | null;
raw: Buffer;
handled: boolean;
},
) {
if (key !== 'email.ingress.feedback') return;
state.feedback.push(data);
if (state.takeFeedback) data.handled = true;
},
},
};
}
if (layer === 'store') {
return {
user: {
getByUsername: async (username: string) =>
state.users.get(username) ?? null,
},
};
}
return {
fs: {
async write(
userId: number,
{
fileMetadata,
fileContent,
}: {
fileMetadata: { path: string };
fileContent: unknown;
},
_uploadTracker?: unknown,
_storageAllowanceMax?: number,
homeRegion?: string,
) {
state.writes.push({
userId,
path: fileMetadata.path,
homeRegion,
content: fileContent,
});
return { fsEntry: { path: fileMetadata.path } };
},
},
};
},
},
}));
import './index.js';
const permanent = (id: number, username: string) => ({
id,
username,
email: `${username}@example.com`,
password: '$2b$10$hash',
});
/** No password and no email is what makes an account temporary. */
const temp = (id: number, username: string) => ({
id,
username,
email: null,
password: null,
});
const ingress = async (
to: string,
over: { secret?: string; contentLength?: string | null } = {},
) => {
const route = state.routes.find((r) => r.path === '/email/ingress');
if (!route) throw new Error('route not registered');
const body = Buffer.from('From: [email protected]\r\n\r\nhello\r\n');
const req = Readable.from([body], { objectMode: false }) as unknown as {
headers: Record<string, string>;
query: Record<string, string>;
};
// `contentLength: null` sends no length header at all, which is the one
// framing the HTTP parser does not bound for us.
req.headers =
over.contentLength === null
? {}
: {
'content-length': over.contentLength ?? String(body.length),
};
req.query = {
SECRET: over.secret ?? SECRET,
to,
from: '[email protected]',
subject: 'hi',
};
let status = 200;
let ended = false;
// `end` takes (chunk, encoding) and no callback, the way the compression
// middleware leaves it, and emits `finish` — which is what the refusal
// paths hang their drain/destroy on. A stub that accepts a callback
// instead hides the bug that took the 413 path to production as a 500.
const emitter = new EventEmitter();
const res = {
status(code: number) {
status = code;
return res;
},
end(chunk?: unknown, encoding?: BufferEncoding) {
if (chunk) Buffer.byteLength(chunk as string, encoding);
ended = true;
emitter.emit('finish');
},
once(event: string, listener: () => void) {
emitter.once(event, listener);
return res;
},
} as unknown as Response;
await route.handler(req as unknown as Request, res);
return {
status,
ended,
drained: (req as unknown as Readable).readableFlowing,
destroyed: (req as unknown as Readable).destroyed,
};
};
beforeEach(() => {
state.users.clear();
state.writes.length = 0;
state.feedback.length = 0;
state.takeFeedback = true;
});
describe('temporary accounts cannot receive', () => {
test('mail for a temp account is accepted and dropped', async () => {
state.users.set('tempy', temp(1, 'tempy'));
const { status, ended, drained } = await ingress('[email protected]');
// Accepted, so the relay neither retries nor bounces to the sender.
expect(status).toBe(200);
expect(ended).toBe(true);
// The body is read off the wire rather than left unconsumed.
expect(drained).toBe(true);
expect(state.writes).toEqual([]);
});
test('mail for a permanent account still lands in the mailbox', async () => {
state.users.set('bob', permanent(2, 'bob'));
const { status } = await ingress('[email protected]');
expect(status).toBe(200);
expect(state.writes).toHaveLength(1);
expect(state.writes[0].userId).toBe(2);
expect(state.writes[0].path).toMatch(/^\/bob\/\.mail\/objects\//);
});
test('an unknown address is still rejected outright', async () => {
const { status } = await ingress('[email protected]');
expect(status).toBe(404);
expect(state.writes).toEqual([]);
});
});
describe('inbound mail is placed in the recipient home region', () => {
/** A permanent account with explicit placement columns. */
const placed = (
id: number,
username: string,
over: { home?: string | null; signup_server?: string | null },
) => ({
...permanent(id, username),
home: null,
signup_server: null,
...over,
});
test("uses the recipient's own home region when they have one", async () => {
state.users.set('bob', placed(2, 'bob', { home: 'frankfurt' }));
await ingress('[email protected]');
expect(state.writes[0].homeRegion).toBe('frankfurt');
});
test('falls back to the server that served their signup', async () => {
// Every account that predates the `home` column is this case.
state.users.set(
'bob',
placed(2, 'bob', { home: null, signup_server: 'london' }),
);
await ingress('[email protected]');
expect(state.writes[0].homeRegion).toBe('london');
});
test('falls back to the primary region when neither is set', async () => {
state.users.set('bob', placed(2, 'bob', {}));
await ingress('[email protected]');
expect(state.writes[0].homeRegion).toBe('oregon');
});
test('home wins over signup_server when both are set', async () => {
state.users.set(
'bob',
placed(2, 'bob', { home: 'sydney', signup_server: 'london' }),
);
await ingress('[email protected]');
expect(state.writes[0].homeRegion).toBe('sydney');
});
});
describe('refusals dispose of the body nobody read', () => {
test('an unknown recipient drains it and keeps the connection', async () => {
// The secret checked out and the length is under the cap, so the body
// was ours to read: discard it rather than resetting a usable socket.
const { status, drained, destroyed } =
await ingress('[email protected]');
expect(status).toBe(404);
expect(drained).toBe(true);
expect(destroyed).toBe(false);
});
test('a bad secret closes rather than reading the upload', async () => {
state.users.set('bob', permanent(2, 'bob'));
const { status, destroyed } = await ingress('[email protected]', {
secret: 'not-the-secret',
});
expect(status).toBe(403);
// Nothing authenticated this caller; their bytes are not worth reading.
expect(destroyed).toBe(true);
expect(state.writes).toEqual([]);
});
test('no content-length is refused with 411', async () => {
state.users.set('bob', permanent(2, 'bob'));
const { status, destroyed } = await ingress('[email protected]', {
contentLength: null,
});
expect(status).toBe(411);
expect(destroyed).toBe(true);
expect(state.writes).toEqual([]);
});
test('a declared length over the cap is refused with 413', async () => {
state.users.set('bob', permanent(2, 'bob'));
const { status, destroyed } = await ingress('[email protected]', {
contentLength: String(25 * 1024 * 1024 + 1),
});
expect(status).toBe(413);
expect(destroyed).toBe(true);
expect(state.writes).toEqual([]);
});
});
describe('feedback addresses', () => {
test('are handed to a listener and reach no mailbox', async () => {
const { status, ended } = await ingress('[email protected]');
expect(status).toBe(200);
expect(ended).toBe(true);
expect(state.feedback).toHaveLength(1);
expect(state.feedback[0].to).toBe('[email protected]');
expect(state.feedback[0].from).toBe('[email protected]');
expect(state.feedback[0].raw.toString()).toContain('hello');
expect(state.writes).toEqual([]);
});
test('match the local part case-insensitively', async () => {
await ingress('[email protected]');
expect(state.feedback).toHaveLength(1);
});
test('fall through to normal routing when nobody takes them', async () => {
state.takeFeedback = false;
state.users.set('abuse', permanent(3, 'abuse'));
const { status } = await ingress('[email protected]');
expect(status).toBe(200);
expect(state.feedback).toHaveLength(1);
// The body was already read, so the buffered copy is what is stored.
expect(state.writes).toHaveLength(1);
expect(Buffer.isBuffer(state.writes[0].content)).toBe(true);
});
test('an unclaimed one with no such account is still a 404', async () => {
state.takeFeedback = false;
const { status } = await ingress('[email protected]');
expect(status).toBe(404);
});
test('only count on a Puter mail domain', async () => {
state.users.set('abuse', permanent(3, 'abuse'));
await ingress('[email protected]');
expect(state.feedback).toEqual([]);
});
test('are refused over the feedback size cap without being read', async () => {
const { status, destroyed } = await ingress('[email protected]', {
contentLength: String(5 * 1024 * 1024 + 1),
});
expect(status).toBe(413);
expect(destroyed).toBe(true);
expect(state.feedback).toEqual([]);
});
});
+161
View File
@@ -0,0 +1,161 @@
import { extension } from '@heyputer/backend/src/extensions';
import { createHmac, randomBytes, timingSafeEqual } from 'node:crypto';
import type { Request } from 'express';
import {
MAX_MESSAGE_BYTES,
isPuterEmailAddress,
isTempUser,
puterEmailUsername,
storeInboxMessage,
} from '@heyputer/backend/src/services/email/mailbox.js';
// copied form peer secrets checker
const COMPARE_KEY = randomBytes(32);
const secretsEqual = (a: string, b: string): boolean =>
timingSafeEqual(
createHmac('sha256', COMPARE_KEY).update(a).digest(),
createHmac('sha256', COMPARE_KEY).update(b).digest(),
);
const USER_EMAIL_CONFIG = (extension.config as Record<string, unknown>)
.userEmail as { secret?: string; feedbackAddresses?: unknown } | undefined;
const INGRESS_SECRET = USER_EMAIL_CONFIG?.secret;
/**
* Local parts that take complaint feedback rather than naming a mailbox. The
* same names are reserved at signup, so no account can be handed them.
*/
const DEFAULT_FEEDBACK_ADDRESSES = ['fbl', 'abuse', 'postmaster'];
const FEEDBACK_ADDRESSES = new Set(
(Array.isArray(USER_EMAIL_CONFIG?.feedbackAddresses)
? USER_EMAIL_CONFIG.feedbackAddresses
: DEFAULT_FEEDBACK_ADDRESSES
).map((local) => String(local).toLowerCase()),
);
/**
* Feedback is handed over whole, so it is read into memory; a complaint report
* quotes one message and has no business near the mailbox ceiling.
*/
const MAX_FEEDBACK_BYTES = 5 * 1024 * 1024;
const isFeedbackAddress = (to: string): boolean =>
isPuterEmailAddress(to) &&
FEEDBACK_ADDRESSES.has(puterEmailUsername(to).toLowerCase());
const readBody = async (req: Request): Promise<Buffer> => {
const chunks: Buffer[] = [];
for await (const chunk of req) chunks.push(chunk as Buffer);
return Buffer.concat(chunks);
};
if (!INGRESS_SECRET) {
console.warn(
'[useremailrx] config.userEmail.secret unset - email ingress disabled',
);
} else {
extension.post(
'/email/ingress',
{
subdomain: 'api',
},
async (req, res) => {
// Answer first, then deal with the body nobody read. Touching the
// request before the response has flushed takes the response down
// with the connection and the caller sees a reset with no status,
// so this hangs off `finish` — and it has to be the event rather
// than `res.end(cb)`, because the compression middleware replaces
// `res.end` with an `(chunk, encoding)` signature that has no
// callback parameter, so a callback lands in `chunk` and dies in
// `Buffer.byteLength`.
//
// `drain` discards the rest of a body we were entitled to read and
// leaves the connection reusable. `close` is for bodies we do not
// want to read at all: an unauthenticated caller's, or one whose
// declared length is missing or over the cap.
const refuse = (status: number, how: 'drain' | 'close') => {
res.once('finish', () =>
how === 'drain' ? req.resume() : req.destroy(),
);
res.status(status).end();
};
const presented = req.query.SECRET;
if (
typeof presented !== 'string' ||
!secretsEqual(presented, INGRESS_SECRET)
) {
// Nothing has authenticated this caller, so refuse to spend
// bandwidth reading whatever they were uploading.
refuse(403, 'close');
return;
}
const size = Number(req.headers['content-length']);
if (!Number.isInteger(size) || size <= 0) {
refuse(411, 'close');
return;
}
if (size > MAX_MESSAGE_BYTES) {
refuse(413, 'close');
return;
}
const stores = extension.import('store');
const services = extension.import('service');
const to = String(req.query.to ?? '');
let content: Buffer | Request = req;
if (isFeedbackAddress(to)) {
if (size > MAX_FEEDBACK_BYTES) {
refuse(413, 'close');
return;
}
const raw = await readBody(req);
const event = {
to,
from:
typeof req.query.from === 'string'
? req.query.from
: null,
raw,
handled: false,
};
await extension
.import('client')
.event.emitAndWait('email.ingress.feedback', event, {});
if (event.handled) {
res.end();
return;
}
// Nobody took it: route it like any other message.
content = raw;
}
const user = await stores.user.getByUsername(
puterEmailUsername(to),
);
if (!user) {
refuse(404, 'drain');
return;
}
if (isTempUser(user)) {
if (content === req) req.resume();
res.end();
return;
}
await storeInboxMessage(services.fs, user, {
subject: req.query.subject,
content,
size: Buffer.isBuffer(content) ? content.length : size,
});
const { SECRET: _secret, ..._loggableQuery } = req.query;
// console.log('got email: ', puterUser, loggableQuery, size);
res.end();
},
);
}
+12
View File
@@ -0,0 +1,12 @@
{
"name": "emailrx",
"version": "1.0.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "emailrx",
"version": "1.0.0"
}
}
}
+6
View File
@@ -0,0 +1,6 @@
{
"name": "emailrx",
"version": "1.0.0",
"main": "index.ts",
"description": "Internal service which handles routing puter emails to users"
}
+187 -26
View File
@@ -1,14 +1,8 @@
import type { Request, Response } from 'express';
import { installSeatStub } from '@heyputer/backend/testFixtures/seatStub';
import { v4 as uuidv4 } from 'uuid';
import {
afterAll,
beforeAll,
describe,
expect,
it,
vi,
} from 'vitest';
import { makeActor } from '../src/backend/core/actor.ts';
import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest';
import { makeActor, type Actor } from '../src/backend/core/actor.ts';
import { runWithContext } from '../src/backend/core/context.ts';
import { configContainer } from '../src/backend/exports.ts';
import { PuterServer } from '../src/backend/server.ts';
@@ -49,7 +43,6 @@ beforeAll(async () => {
create_shortcut: true,
payment_bypass: true,
},
teams_enabled: true,
} as never);
});
@@ -83,21 +76,20 @@ describe('whoami extension — handleWhoami', () => {
};
const seatOf = async (teamName: string) => {
const owner = await seedUser();
const seat = await seedSeat();
const team = await server.stores.team.create({
ownerUserId: owner.id as number,
const team = {
uid: `wt-${Math.random().toString(36).slice(2, 9)}`,
name: teamName,
handle: `wt-${Math.random().toString(36).slice(2, 9)}`,
});
await server.stores.team.addMember(team.uid, seat.id as number, {
orgOwned: true,
};
installSeatStub(server as never, [seat.id as number], {
teamUid: team.uid,
teamName: team.name,
});
return { seat, team };
};
it('names the team for a seat', async () => {
const { seat, team } = await seatOf('Acme Corp');
it('names no team where teams is off', async () => {
const { seat } = await seatOf('Acme Corp');
const { res, captured } = makeRes();
await runWithContext(
@@ -105,10 +97,8 @@ describe('whoami extension — handleWhoami', () => {
() => handleWhoami(makeReq(), res),
);
expect((captured.body as { team?: unknown }).team).toEqual({
uid: team.uid,
name: 'Acme Corp',
});
// Off, so the seat is never looked up.
expect((captured.body as { team?: unknown }).team).toBeUndefined();
});
it('says nothing for an account that is not a seat', async () => {
@@ -170,6 +160,81 @@ describe('whoami extension — handleWhoami', () => {
expect(captured.body).toEqual({ error: 'User not found' });
});
describe("an app's view of the email", () => {
const appActorFor = async (user: { uuid: string; id: unknown }) => {
const app = await (
server.stores.app.create as unknown as (
fields: Record<string, unknown>,
opts: { ownerUserId: number },
) => Promise<{ uid: string; id: number }>
)(
{
name: `we-${uuidv4()}`,
title: 'Email reader',
index_url: `https://${uuidv4()}.example/`,
},
{ ownerUserId: user.id as number },
);
return {
app,
actor: makeActor({
user: { uuid: user.uuid, id: user.id as number },
app: { uid: app.uid, id: app.id },
}),
};
};
const whoamiAs = async (actor: Actor) => {
const { res, captured } = makeRes();
await runWithContext({ actor }, () =>
handleWhoami(makeReq(), res),
);
return captured;
};
it('is withheld until the user grants it, then shown', async () => {
const user = await seedUser();
const { app, actor } = await appActorFor(user);
const before = (await whoamiAs(actor)).body as Record<
string,
unknown
>;
expect(before).not.toHaveProperty('email');
expect(before).not.toHaveProperty('unconfirmed_email');
await server.services.permission.grantUserAppPermission(
makeActor({ user: { uuid: user.uuid, id: user.id as number } }),
app.uid,
`user:${user.uuid}:email:read`,
);
const after = (await whoamiAs(actor)).body as Record<
string,
unknown
>;
expect(after.email).toBe(user.email);
expect(after.unconfirmed_email).toBe(user.email);
});
it('answers 404 for a missing account even when the check fails', async () => {
const check = vi
.spyOn(server.services.permission, 'check')
.mockRejectedValue(new Error('scan failed'));
try {
const captured = await whoamiAs(
makeActor({
user: { uuid: 'ghost-uuid', id: 99_999_998 },
app: { uid: 'app-ghost', id: 1 },
}),
);
expect(captured.statusCode).toBe(404);
expect(captured.body).toEqual({ error: 'User not found' });
} finally {
check.mockRestore();
}
});
});
it('returns full user details for a user actor', async () => {
const user = await seedUser();
const { res, captured } = makeRes();
@@ -285,6 +350,104 @@ describe('whoami extension — handleWhoami', () => {
expect(body.directories).toBeUndefined();
});
describe('fields added by whoami.details listeners', () => {
const addAccountDetails = (
_key: unknown,
event: { details: Record<string, unknown> },
) => {
Object.assign(event.details, {
subscribed: true,
paid_storage: 1024,
subscription: { tier: 'pro', status: 'trialing' },
accountEligible: true,
});
};
const whoamiWithListener = async (
actor: Actor,
listener: typeof addAccountDetails = addAccountDetails,
) => {
const { res, captured } = makeRes();
server.clients.event.on('whoami.details', listener);
try {
await runWithContext({ actor }, () =>
handleWhoami(makeReq(), res),
);
} finally {
server.clients.event.off('whoami.details', listener);
}
return captured.body as Record<string, unknown>;
};
it('reach an app actor only as the documented subscribed / paid_storage', async () => {
const user = await seedUser();
const body = await whoamiWithListener(
makeActor({
user: { uuid: user.uuid, id: user.id as number },
app: { uid: 'app-test-actor' },
}),
);
expect(body.subscribed).toBe(true);
expect(body.paid_storage).toBe(1024);
expect(body).not.toHaveProperty('subscription');
expect(body).not.toHaveProperty('accountEligible');
// Fields the handler built itself still ship.
expect(body.username).toBe(user.username);
expect(body.app_name).toBe('app-test-actor');
});
it('are withheld from an app reached through an access token it issued', async () => {
const user = await seedUser();
const owner = { uuid: user.uuid, id: user.id as number };
const body = await whoamiWithListener(
makeActor({
user: owner,
accessToken: {
uid: 'tok-1',
issuer: makeActor({
user: owner,
app: { uid: 'app-issuer' },
}),
},
}),
);
expect(body.subscribed).toBe(true);
expect(body).not.toHaveProperty('subscription');
expect(body).not.toHaveProperty('accountEligible');
});
it('all reach a user actor', async () => {
const user = await seedUser();
const body = await whoamiWithListener(
makeActor({ user: { uuid: user.uuid, id: user.id as number } }),
);
expect(body).toMatchObject({
subscribed: true,
paid_storage: 1024,
subscription: { tier: 'pro', status: 'trialing' },
accountEligible: true,
});
});
it('cannot add taskbar_items for an app actor, which core omits', async () => {
const user = await seedUser();
const body = await whoamiWithListener(
makeActor({
user: { uuid: user.uuid, id: user.id as number },
app: { uid: 'app-test-actor' },
}),
(_key, event) => {
event.details.taskbar_items = [{ name: 'injected' }];
},
);
expect(body).not.toHaveProperty('taskbar_items');
});
});
it('redacts tmp_password from metadata for user actors', async () => {
const user = await seedUser();
await server.stores.user.updateMetadata(user.id as number, {
@@ -353,9 +516,7 @@ describe('whoami extension — handleWhoami', () => {
}),
]) {
const { res, captured } = makeRes();
await runWithContext({ actor }, () =>
handleWhoami(makeReq(), res),
);
await runWithContext({ actor }, () => handleWhoami(makeReq(), res));
const body = captured.body as Record<string, unknown>;
expect(body.phone).toBeUndefined();
+48 -229
View File
@@ -1,11 +1,10 @@
import { Context } from '@heyputer/backend/src/core';
import { extension } from '@heyputer/backend/src/extensions';
import {
cardFallbackDepsFrom,
isCardFallbackEligible,
} from '@heyputer/backend/src/util/cardFallback.js';
import { APP_ICON_SIZES } from '@heyputer/backend/src/util/appIcon.js';
import { getTaskbarItems } from '@heyputer/backend/src/util/taskbarItems.js';
import {
buildUserDetails,
scrubSensitive,
} from '@heyputer/backend/src/util/userDetails.js';
import type { Request, Response } from 'express';
import TimeAgo from 'javascript-time-ago';
import localeEn from 'javascript-time-ago/locale/en';
@@ -19,87 +18,13 @@ const timeago = (() => {
return new TimeAgo('en-US');
})();
// User timestamps come off the DB as SQL datetime strings; the wire format
// for all of them is unix seconds. Unparseable values are dropped rather
// than sent as NaN.
const toUnixSeconds = (value: unknown): number | undefined => {
if (!value) return undefined;
const ms = new Date(value as string | number | Date).getTime();
return Number.isNaN(ms) ? undefined : Math.round(ms / 1000);
};
// Allowlist of `config.feature_flags` keys safe to surface via /whoami.
// Anything not listed here stays server-side, so internal flags
// (payment_bypass, staff_only_*, etc.) cannot leak by accident. Add a
// flag here when, and only when, the client actually needs to read it.
const CLIENT_VISIBLE_FEATURE_FLAGS: ReadonlySet<string> = new Set([
'create_shortcut',
'download_directory',
'prompt_user_when_navigation_away_from_puter',
// What an app actor may receive from `whoami.details` listeners. Anything
// else a listener adds describes the account, and only ships to user actors.
const APP_VISIBLE_LISTENER_KEYS: ReadonlySet<string> = new Set([
'subscribed',
'paid_storage',
]);
// Keys that must never leave the server, whoever put them on the response.
// `details` is an explicit pick, but the `whoami.details` event hands
// listeners the full UserRow next to the object they may write to, and
// `metadata` is a free-form blob — so any of these can arrive on the
// response without an edit to the pick above. The scrub runs last, over the
// whole payload, and is the one place that decides what "sensitive" means.
//
// Credentials and single-use tokens, the payment/phone identifiers used for
// verification (`card_fingerprint` is the Stripe fingerprint, stable per
// card number), the network identity recorded at signup, and internal
// anti-abuse bookkeeping. `requires_phone_verification` /
// `requires_card_verification` stay: they are the flags the GUI acts on, and
// they carry no identifier.
const SENSITIVE_KEYS: ReadonlySet<string> = new Set([
'password',
'tmp_password',
'pass_recovery_token',
'email_confirm_code',
'email_confirm_token',
'change_email_confirm_token',
'otp_secret',
'otp_recovery_codes',
'card_fingerprint',
'phone',
'clean_email',
'signup_ip',
'signup_ip_forwarded',
'signup_user_agent',
'signup_origin',
'signup_server',
'audit_metadata',
]);
// Depth-limited, cycle-safe walk deleting every SENSITIVE_KEYS entry it finds
// at any level (`metadata` and `taskbar_items` are both nested structures).
const scrubSensitive = (
value: unknown,
seen: Set<object> = new Set(),
depth = 0,
): void => {
if (depth > 8 || value === null || typeof value !== 'object') return;
if (seen.has(value as object)) return;
seen.add(value as object);
if (Array.isArray(value)) {
for (const entry of value) scrubSensitive(entry, seen, depth + 1);
return;
}
for (const key of Object.keys(value as Record<string, unknown>)) {
if (SENSITIVE_KEYS.has(key)) {
delete (value as Record<string, unknown>)[key];
continue;
}
scrubSensitive(
(value as Record<string, unknown>)[key],
seen,
depth + 1,
);
}
};
export const handleWhoami = async (
req: Request,
res: Response,
@@ -111,13 +36,24 @@ export const handleWhoami = async (
}
const isUser = !actor.effectiveApp;
const canReadEmail = async (uuid: unknown) => {
try {
return await services.permission.check(
actor,
`user:${uuid}:email:read`,
);
} catch {
return false;
}
};
// Started beside the row read: the actor carries the row's uuid.
const emailCheck = isUser ? null : canReadEmail(actor.user.uuid);
const user = await stores.user.getById(actor.user.id);
if (!user) {
res.status(404).json({ error: 'User not found' });
return;
}
const oidcOnly = user.password === null;
const rawIconSize =
typeof req.query?.icon_size === 'string'
? Number(req.query.icon_size)
@@ -126,153 +62,22 @@ export const handleWhoami = async (
rawIconSize !== undefined && APP_ICON_SIZES.includes(rawIconSize)
? rawIconSize
: undefined;
const noIcons = !iconSize;
// Feature flags come from `config.feature_flags`. We only forward keys
// listed in CLIENT_VISIBLE_FEATURE_FLAGS so internal flags can't leak.
// Non-boolean values (e.g. `"true"` as a string) are coerced so the
// client never has to guess.
const rawFlags = extension.config.feature_flags ?? {};
const feature_flags: Record<string, boolean> = {};
for (const [k, v] of Object.entries(rawFlags)) {
if (CLIENT_VISIBLE_FEATURE_FLAGS.has(k)) {
feature_flags[k] = Boolean(v);
}
}
// Deep-copied (it is decoded JSON) so the scrub below edits the response
// and not the cached UserRow. Sensitive keys inside it — tmp_password and
// anything else on the denylist — are removed by scrubSensitive.
const metadata = user.metadata
? structuredClone(user.metadata)
: user.metadata;
const details: Record<string, unknown> = {
username: user.username,
uuid: user.uuid,
email: user.email,
unconfirmed_email: user.email,
email_confirmed: user.email_confirmed || user.username === 'admin',
requires_email_confirmation: user.requires_email_confirmation,
// The phone number itself is deliberately absent: nothing on the
// client reads it, and it is PII that would otherwise be handed to
// every app actor. Only the verification flag ships.
requires_phone_verification: user.requires_phone_verification,
requires_card_verification: user.requires_card_verification,
// A seat reaches nothing until it replaces its admin's password.
requires_password_change: user.requires_password_change,
// The SMS-to-card escape hatch: true once this user is out of SMS send
// attempts and may verify a card instead. It has to ship from here
// because /send-confirm-phone can no longer say so — by the time the
// fallback opens, further sends are rejected by that route's own rate
// limit before any handler runs, so a page reload would otherwise lose
// an offer that stays valid for 24 hours. Only for user actors, and it
// costs no KV read unless the account is actually phone-gated.
card_fallback_available: isUser
? await isCardFallbackEligible(
extension.config,
user,
async (key) => (await stores.kv.get({ key })).res,
cardFallbackDepsFrom(clients),
)
: false,
desktop_bg_url: user.desktop_bg_url,
desktop_bg_color: user.desktop_bg_color,
desktop_bg_fit: user.desktop_bg_fit,
is_temp: user.password === null && user.email === null,
is_user_token: true,
// Present only once the account has actually asked for a code (see the
// referral extension) — null until then, and never minted from here:
// this endpoint is polled, and a mint is a write.
referral_code: user.referral_code,
oidc_only: oidcOnly,
taskbar_items: isUser
? await getTaskbarItems(
user,
{
clients,
stores,
services,
apiBaseUrl: String(extension.config.api_base_url ?? ''),
config: extension.config,
},
{ iconSize, noIcons },
)
: undefined,
otp: !!user.otp_enabled,
feature_flags,
created_ts: toUnixSeconds(user.timestamp),
human_readable_age: user.timestamp
? timeago.format(new Date(user.timestamp as string))
: null,
metadata,
hasDevAccountAccess: !!user.metadata?.hasDevAccountAccess,
};
// OIDC revalidate URL for password-less accounts
if (oidcOnly) {
try {
const provider = await services.oidc.getLinkedProviderForUser(
user.id as number,
);
if (provider) {
const origin = (extension.config.origin ?? '').replace(
/\/$/,
'',
);
details.oidc_revalidate_url = `${origin}/auth/oidc/${provider}/start?flow=revalidate&user_uuid=${encodeURIComponent(user.uuid)}`;
}
} catch {
// OIDC not configured
}
}
// Directories — only sent to user actors
if (isUser) {
const directories: Record<string, unknown> = {};
const nameToProp: Record<string, string> = {
desktop_uuid: `/${user.username}/Desktop`,
appdata_uuid: `/${user.username}/AppData`,
documents_uuid: `/${user.username}/Documents`,
pictures_uuid: `/${user.username}/Pictures`,
videos_uuid: `/${user.username}/Videos`,
trash_uuid: `/${user.username}/Trash`,
};
for (const k in nameToProp) {
directories[nameToProp[k]] = user[k];
}
details.directories = directories;
}
// The team an account belongs to, when it is one a team pays for. User
// actors only, and only where teams are on.
if (isUser && extension.config.teams_enabled === true) {
try {
const seat = await stores.team.getOrgSeat(user.id);
if (seat) {
details.team = {
uid: seat.team_uid,
name: seat.team_name ?? null,
};
}
} catch (e) {
// Never fail whoami over this; the account still works without it.
console.warn('[whoami] team lookup failed:', (e as Error).message);
}
}
// Last activity
const lastActivityTs = toUnixSeconds(user.last_activity_ts);
if (lastActivityTs !== undefined) {
details.last_activity_ts = lastActivityTs;
}
const details = await buildUserDetails(
user,
{ config: extension.config, clients, stores, services },
{ isUser, iconSize, noIcons: !iconSize },
);
details.human_readable_age = user.timestamp
? timeago.format(new Date(user.timestamp as string))
: null;
// Strip sensitive fields for app actors
if (!isUser) {
const canReadEmail = await services.permission
.check(actor, `user:${user.uuid}:email:read`)
.catch(() => false);
if (!canReadEmail) {
const emailReadable = await (user.uuid === actor.user.uuid
? emailCheck
: canReadEmail(user.uuid));
if (!emailReadable) {
delete details.email;
delete details.unconfirmed_email;
}
@@ -292,6 +97,12 @@ export const handleWhoami = async (
details.app_name = app.uid;
}
// A key core left undefined isn't built, so a listener can't fill it for an app.
const builtKeys = isUser
? null
: new Set(
Object.keys(details).filter((key) => details[key] !== undefined),
);
try {
await clients.event.emitAndWait(
'whoami.details',
@@ -301,9 +112,17 @@ export const handleWhoami = async (
} catch {
/* best-effort */
}
if (builtKeys) {
for (const key of Object.keys(details)) {
if (!builtKeys.has(key) && !APP_VISIBLE_LISTENER_KEYS.has(key)) {
delete details[key];
}
}
}
const subscription = details.subscription as
{ offering?: Record<string, unknown> } | undefined;
| { offering?: Record<string, unknown> }
| undefined;
if (subscription?.offering) {
delete subscription.offering.group;
delete subscription.offering.benefits;
+836 -174
View File
File diff suppressed because it is too large. Load diff
+3 -2
View File
@@ -50,6 +50,7 @@
"build:workerLib:coverage": "cd src/puter-js && npm run build:coverage && cd ../worker && npm run build",
"start:gui": "nodemon --exec \"node dev-server.js\" ",
"start": "node ./tools/start.mjs",
"start:smtp": "node ./tools/start.mjs --smtp",
"dev": "npm start",
"build": "npm run setupExtensions && npm run build:ts && cd src/gui && node ./build.js && cd ../puter-js && npm run build",
"build:workerLib": "cd src/puter-js && npm run build && cd ../worker && npm run build",
@@ -76,7 +77,7 @@
"@aws-sdk/s3-request-presigner": "^3.1028.0",
"dedent": "^1.5.3",
"javascript-time-ago": "^2.5.11",
"libphonenumber-js": "1.13.13",
"libphonenumber-js": "1.13.15",
"miniflare": "^4.20260617.1",
"open": "^10.1.0"
},
@@ -96,6 +97,6 @@
"workerd": true
},
"overrides": {
"undici": "^7.29.0"
"undici": "^7.30.0"
}
}
+135
View File
@@ -0,0 +1,135 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { v4 as uuidv4 } from 'uuid';
import { afterAll, beforeAll, describe, expect, it } from 'vitest';
import type { PuterServer } from './server.ts';
import { createTestUser, setupTestServer } from './testUtil.ts';
/**
* Teams is optional, so core has to work when neither the service nor the store
* is registered, which is how this test server boots.
*/
describe('core without a teams registration', () => {
let server: PuterServer;
const makeUser = async (): Promise<{ id: number; username: string }> => {
const username = `nt_${Math.random().toString(36).slice(2, 10)}`;
const created = (await server.stores.user.create({
username,
uuid: uuidv4(),
password: null,
email: `${username}@test.local`,
})) as unknown as { id: number };
return { id: created.id, username };
};
beforeAll(async () => {
server = await setupTestServer({
// Without this the allowlist check returns before asking Teams.
teams_allowed_email_domains: ['test.local'],
} as never);
});
afterAll(async () => {
await server?.shutdown();
});
it('has no teams in the registry', () => {
expect(server.services.team).toBeUndefined();
expect(server.stores.team).toBeUndefined();
});
it('refuses to boot when the config asks for teams', async () => {
await expect(
setupTestServer({ teams_enabled: true } as never),
).rejects.toThrow(/teams_enabled/);
});
it('deletes an account without asking what it was charged for', async () => {
const user = await makeUser();
await expect(
server.services.userAccount.cascadeDelete(user.id),
).resolves.not.toThrow();
expect(await server.stores.user.getById(user.id)).toBeFalsy();
});
it('answers the address-claim question signup asks', async () => {
const user = await makeUser();
const row = (await server.stores.user.getById(user.id))!;
// Nothing is a seat, so the holder is returned as-is.
await expect(
server.services.signup.signupEmailHolder(row.email as string, {
releaseSeat: true,
}),
).resolves.toMatchObject({ id: user.id });
});
it('lists what is shared with an account, folding in no team shares', async () => {
const user = await makeUser();
await expect(
server.services.share.listSharedWithMe({
user: { id: user.id },
} as never),
).resolves.toMatchObject({ items: [] });
});
it('refuses a share aimed at a team, rather than failing', async () => {
// A real home directory, so the refusal is the recipient's and not
// the entry's.
const username = `nt_${Math.random().toString(36).slice(2, 10)}`;
const creds = await createTestUser(server, {
username,
password: 'Nt-Pw-12345!',
});
const row = (await server.stores.user.getByUsername(username))!;
const user = { id: row.id, username };
const actor = { user: row } as never;
expect(creds).toBeTruthy();
// The recipient cannot exist, so this is a 404 and not a crash.
await expect(
server.services.share.share(actor, {
recipient: { teamHandle: 'nobody' },
mode: 'read',
path: `/${user.username}`,
} as never),
).rejects.toMatchObject({ statusCode: 404 });
});
it('serves the homepage shell with the teams tab off', async () => {
const user = await makeUser();
const row = (await server.stores.user.getById(user.id))!;
let html = '';
const res = { send: (out: string) => (html = out) };
await server.services.homepage.send(
{
req: { headers: {}, query: {}, get: () => undefined },
res,
actor: { user: row },
} as never,
{ title: 'Puter' } as never,
{} as never,
);
const params = JSON.parse(
/gui\((\{.*?\})\);/s.exec(html)![1].replaceAll('\\u003c', '<'),
);
expect(params.teams_ui).toBe(false);
});
});
+123 -2
View File
@@ -1,5 +1,9 @@
import { beforeEach, describe, expect, it, vi } from 'vitest';
import { AlarmClient } from './AlarmClient';
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import {
ALARM_IDLE_TTL_MS,
AlarmClient,
MAX_TRACKED_ALARMS,
} from './AlarmClient';
import type { IConfig, IPagerConfig } from '../../types';
import type { AlertPayload } from './types';
@@ -342,6 +346,19 @@ describe('AlarmClient alarm registry', () => {
expect(client.get(shortId)).toBeUndefined();
});
it('keeps the short id of another alarm that shares it when one is cleared', () => {
const client = makeClient();
// These two ids hash to the same short id.
client.create('alarm:274', 'first');
const shortId = client.get('alarm:274')!.shortId;
client.create('alarm:643', 'second');
expect(client.get('alarm:643')!.shortId).toBe(shortId);
client.clear('alarm:274');
expect(client.get(shortId)?.id).toBe('alarm:643');
});
it('ignores a clear for an alarm that is not active', () => {
const client = makeClient();
expect(() => client.clear('never-raised')).not.toThrow();
@@ -428,6 +445,110 @@ describe('AlarmClient alarm registry', () => {
});
});
describe('AlarmClient alarm eviction', () => {
beforeEach(() => {
vi.useFakeTimers();
vi.spyOn(console, 'error').mockImplementation(() => {});
vi.spyOn(console, 'warn').mockImplementation(() => {});
vi.spyOn(console, 'log').mockImplementation(() => {});
});
afterEach(() => {
vi.useRealTimers();
});
it('forgets alarms that stay quiet past the idle window, under both ids', () => {
const client = makeClient();
const ids = Array.from(
{ length: 50 },
(_, i) => `metering_usage_dropped:user-${i}`,
);
for (const id of ids) client.create(id, 'usage dropped');
const shortIds = ids.map((id) => client.get(id)!.shortId);
vi.advanceTimersByTime(ALARM_IDLE_TTL_MS + 1);
client.create('unrelated', 'the next alarm sweeps the registry');
expect(ids.filter((id) => client.get(id))).toEqual([]);
expect(shortIds.filter((shortId) => client.get(shortId))).toEqual([]);
expect(client.get('unrelated')).toBeDefined();
});
it('treats a recurrence after the idle window as a new alarm', () => {
const client = makeClient();
const seen = capture(client);
client.create('scan:failed', 'first');
client.create('scan:failed', 'second');
vi.advanceTimersByTime(ALARM_IDLE_TTL_MS + 1);
client.create('scan:failed', 'back again');
expect(seen.at(-1)).toMatchObject({ repeatCount: 1, isRepeat: false });
// A fresh start keeps per-occurrence incident keys from colliding.
expect(seen.at(-1)!.dedupKey).not.toBe(seen[0].dedupKey);
});
it('still treats a repeat inside the window as a repeat after a sweep', () => {
const client = makeClient();
const seen = capture(client);
for (let i = 0; i < 10; i++) client.create('flap', 'same fault');
vi.advanceTimersByTime(ALARM_IDLE_TTL_MS - 1000);
client.create('other', 'sweeps the registry');
client.create('flap', 'same fault');
// Occurrence 11 is past the burst, so nothing new goes out for it.
expect(seen.map((alert) => alert.id)).toEqual([
...Array(10).fill('flap'),
'other',
]);
expect(client.get('flap')?.count).toBe(11);
});
it('keeps an alarm that keeps recurring for longer than the window', () => {
const client = makeClient();
for (let i = 0; i < 6; i++) {
client.create('hot', 'still broken');
vi.advanceTimersByTime(ALARM_IDLE_TTL_MS / 2);
client.create(`noise:${i}`, 'sweeps the registry');
}
expect(client.get('hot')?.count).toBe(6);
});
it('drops the least recently raised alarm once full', () => {
const client = makeClient();
for (let i = 0; i < MAX_TRACKED_ALARMS; i++) {
client.create(`user:${i}`, 'per-user alarm');
vi.advanceTimersByTime(1);
}
// Raised again, so user:1 is now the one seen longest ago.
client.create('user:0', 'per-user alarm');
client.create('one-more', 'over the cap');
expect(client.get('user:1')).toBeUndefined();
expect(client.get('user:0')?.count).toBe(2);
expect(client.get('user:2')).toBeDefined();
expect(client.get('one-more')).toBeDefined();
});
it('never holds more than the cap', () => {
const client = makeClient();
const ids = Array.from(
{ length: MAX_TRACKED_ALARMS * 3 },
(_, i) => `user:${i}`,
);
for (const id of ids) client.create(id, 'per-user alarm');
const held = ids.filter((id) => client.get(id));
expect(held).toHaveLength(MAX_TRACKED_ALARMS);
expect(held[0]).toBe(`user:${MAX_TRACKED_ALARMS * 2}`);
});
});
describe('AlarmClient known-error rules', () => {
beforeEach(() => {
vi.spyOn(console, 'error').mockImplementation(() => {});
+50 -9
View File
@@ -61,14 +61,24 @@ interface RegisteredHandler {
/** Severity used when neither the call site nor config picks one. */
const FALLBACK_SEVERITY: PagerSeverity = 'critical';
/**
* How many recent occurrences an alarm keeps. An alarm is never cleared unless
* something calls `clear`, and a hot one repeats for as long as the fault lasts
* — so retaining every occurrence means retaining every message and field set
* it was ever raised with, request bodies and actors included, for the life of
* the process. The last few are what a human reads; the rest is only a count,
* and `count` keeps that.
* How many recent occurrences an alarm keeps. A hot alarm repeats for as long
* as the fault lasts, so retaining every occurrence means retaining every
* message and field set it was ever raised with, request bodies and actors
* included. The last few are what a human reads; the rest is only a count, and
* `count` keeps that.
*/
const OCCURRENCE_HISTORY_LIMIT = 20;
/**
* An alarm that hasn't recurred for this long is forgotten; its next occurrence
* is a new alarm with a fresh count. Kept far past the chat transport's repeat
* throttle so a slowly recurring fault still counts as one alarm.
*/
export const ALARM_IDLE_TTL_MS = 24 * 60 * 60 * 1000;
/**
* Most alarms held at once. Ids that embed a user or key can outrun the idle
* window; past this the least recently raised alarm is dropped.
*/
export const MAX_TRACKED_ALARMS = 1000;
// A fast-recurring fault repeats far faster than it is useful to report, and
// every report allocates a log line and a payload. Report every occurrence up
@@ -187,6 +197,7 @@ function cleanFields(fields: AlarmFields): Record<string, string> {
* `pager.severityOverrides` in {@link IConfig}.
*/
export class AlarmClient extends PuterClient {
/** Ordered by last occurrence: a repeat moves its alarm to the back. */
private alarms = new Map<string, Alarm>();
private aliases = new Map<string, Alarm>();
private alertHandlers: RegisteredHandler[] = [];
@@ -290,7 +301,8 @@ export class AlarmClient extends PuterClient {
/**
* Create or update an alarm. If the alarm ID already exists, the occurrence
* count is incremented and a repeat alert is dispatched.
* count is incremented and a repeat alert is dispatched. An alarm idle for
* {@link ALARM_IDLE_TTL_MS} is forgotten, so its next occurrence is new.
*
* `severity` decides where the alarm lands:
*
@@ -321,9 +333,12 @@ export class AlarmClient extends PuterClient {
return;
}
this.evict();
const existing = this.alarms.get(id);
if (existing) {
this.alarms.delete(id);
this.alarms.set(id, existing);
this.recordOccurrence(existing, message, fields);
this.handleRepeat(existing);
return;
@@ -348,6 +363,7 @@ export class AlarmClient extends PuterClient {
this.alarms.set(id, alarm);
this.aliases.set(alarm.shortId, alarm);
this.recordOccurrence(alarm, message, fields);
this.evict();
this.handleNew(alarm);
}
@@ -356,8 +372,7 @@ export class AlarmClient extends PuterClient {
const alarm = this.alarms.get(id);
if (!alarm) return;
this.alarms.delete(id);
this.aliases.delete(alarm.shortId);
this.forget(alarm);
console.log(`[alarm] CLEAR ${displayId(alarm)} :: ${alarm.message}`);
}
@@ -395,6 +410,32 @@ export class AlarmClient extends PuterClient {
// -- Internals ----------------------------------------------------
/**
* Drop idle alarms, then the least recently raised ones past the cap. The
* map's order puts both at the front, so this stops at the first keeper.
*/
private evict(): void {
const now = Date.now();
for (const alarm of this.alarms.values()) {
const lastSeen = alarm.timestamps.at(-1) ?? alarm.started;
if (
this.alarms.size <= MAX_TRACKED_ALARMS &&
now - lastSeen < ALARM_IDLE_TTL_MS
) {
return;
}
this.forget(alarm);
}
}
private forget(alarm: Alarm): void {
this.alarms.delete(alarm.id);
// Short ids can collide; leave an alias that now names another alarm.
if (this.aliases.get(alarm.shortId) === alarm) {
this.aliases.delete(alarm.shortId);
}
}
private recordOccurrence(
alarm: Alarm,
message: string,
+7 -22
View File
@@ -18,6 +18,7 @@
*/
import type { ISlackAlertConfig, PagerSeverity } from '../../types';
import { BoundedTtlMap } from '../../util/boundedTtlMap';
import type { AlertHandler, AlertPayload } from './types';
const REQUEST_TIMEOUT_MS = 5000;
@@ -123,30 +124,14 @@ export function createSlackAlertHandler(
): AlertHandler {
const webhookUrl = conf.webhookUrl as string;
const throttleMs = conf.repeatThrottleMs ?? DEFAULT_REPEAT_THROTTLE_MS;
const lastPosted = new Map<string, number>();
const shouldPost = (alert: AlertPayload): boolean => {
if (throttleMs <= 0) return true;
const now = Date.now();
const previous = lastPosted.get(alert.id);
if (previous !== undefined && now - previous < throttleMs) return false;
if (lastPosted.size >= MAX_THROTTLE_ENTRIES) {
for (const [id, at] of lastPosted) {
if (now - at >= throttleMs) lastPosted.delete(id);
}
// Still full of live entries — drop the oldest to stay bounded.
if (lastPosted.size >= MAX_THROTTLE_ENTRIES) {
const oldest = lastPosted.keys().next().value;
if (oldest !== undefined) lastPosted.delete(oldest);
}
}
lastPosted.set(alert.id, now);
return true;
};
// A zero or negative throttle expires every entry on arrival.
const lastPosted = new BoundedTtlMap<string, true>({
maxEntries: MAX_THROTTLE_ENTRIES,
ttlMs: throttleMs,
});
return async (alert) => {
if (!shouldPost(alert)) return;
if (!lastPosted.shouldEmit(alert.id)) return;
const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), REQUEST_TIMEOUT_MS);
@@ -45,10 +45,15 @@ class FixtureDatabaseClient extends AbstractDatabaseClient {
constructor(
private readonly replicaRows: Record<string, unknown>[] | Error,
private readonly primaryRows: Record<string, unknown>[] = [],
private readonly replica = true,
) {
super(config('fixture'));
}
protected override hasReadReplica(): boolean {
return this.replica;
}
override async read(query: string): Promise<Record<string, unknown>[]> {
this.reads.push(query);
if (this.replicaRows instanceof Error) throw this.replicaRows;
@@ -127,14 +132,22 @@ describe('AbstractDatabaseClient — replica-aware reads', () => {
]);
});
it('names the failing query when a required read finds nothing', async () => {
const client = new FixtureDatabaseClient([], []);
await expect(
client.requireRead('SELECT * FROM `user` WHERE `id` = ?'),
).rejects.toThrow(
'required read returned no rows: SELECT * FROM `user` WHERE `id` = ?',
it('surfaces the primary error when both reads fail', async () => {
const client = new FixtureDatabaseClient(new Error('replica down'));
client.pread = async () => {
throw new Error('primary down');
};
await expect(client.tryHardRead('SELECT 1')).rejects.toThrow(
'primary down',
);
});
it('issues a single read when there is no replica', async () => {
const client = new FixtureDatabaseClient([], [{ id: 2 }], false);
await expect(client.tryHardRead('SELECT 1')).resolves.toEqual([]);
expect(client.reads).toEqual(['SELECT 1']);
expect(client.preads).toEqual([]);
});
});
describe('AbstractDatabaseClient — SQL generation', () => {
@@ -235,6 +248,25 @@ describe('AbstractDatabaseClient — SQL generation', () => {
expect(mysql().returningIdClause()).toBe('');
});
it('stamps a timestamp from unix seconds in each dialect', () => {
expect(sqlite().timestampFromUnixSql()).toBe(
"datetime(?, 'unixepoch')",
);
expect(mysql().timestampFromUnixSql()).toBe('FROM_UNIXTIME(?)');
expect(postgres().timestampFromUnixSql()).toBe('to_timestamp(?)');
});
it('describes a stored timestamp the way each driver reads it back', () => {
const seconds = Date.UTC(2026, 9, 11, 8, 5, 9) / 1000;
expect(sqlite().timestampValue(seconds)).toBe('2026-10-11 08:05:09');
expect(mysql().timestampValue(seconds)).toEqual(
new Date(seconds * 1000),
);
expect(postgres().timestampValue(seconds)).toEqual(
new Date(seconds * 1000),
);
});
it('falls back to `otherwise` for an engine with no explicit choice', () => {
const client = new FixtureDatabaseClient([]);
expect(
+189 -19
View File
@@ -17,13 +17,23 @@
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { readdirSync, readFileSync } from 'fs';
import { isAbsolute, resolve as resolvePath } from 'path';
import type { IConfig } from '../../types';
import { Span } from '../../util/span.js';
import { PuterClient } from '../types';
import { compareMigrationFilenames } from './migrationFilenames.js';
export interface WriteResult {
insertId: number | bigint;
affectedRows: number;
anyRowsAffected: boolean;
/**
* Warnings the statement raised; nonzero means a value may have been stored
* differently from how it was sent. Absent when the engine can't vouch that
* a clean insert stored exactly what was sent, on a fresh id.
*/
warningStatus?: number;
}
export interface BatchEntry {
@@ -33,6 +43,12 @@ export interface BatchEntry {
type SqlJsonPath = readonly [string, ...string[]];
/** The slice of a pooled connection that migrations need. */
export interface MigrationConnection {
query(sql: string): Promise<unknown>;
release(): void;
}
/**
* Base database client. Subclasses must override every method that throws here.
*
@@ -104,39 +120,133 @@ export class AbstractDatabaseClient extends PuterClient {
return this.write(sql, values);
}
/** Whether `read()` goes to a separate replica rather than the primary. */
protected hasReadReplica(): boolean {
return false;
}
/**
* Like `read()` but falls back to the primary when read-replicas are in
* use. Subclasses may override with replica-aware logic; the default
* delegates to `pread()`.
* The replica leg of `tryHardRead`. Override when `read()` falls back to
* the primary by itself, or a replica failure reads the primary twice.
*/
protected replicaRead(
query: string,
params: unknown[],
): Promise<Record<string, unknown>[]> {
return this.read(query, params);
}
/**
* Like `read()`, but an empty or failed replica read falls back to the
* primary, so replication lag can't hide a row that was just written. The
* primary read starts alongside the replica one rather than after it.
* Without a replica this is a single `read()`.
*/
@Span('db.tryHardRead', (query: string) => ({ 'db.statement': query }))
async tryHardRead(
query: string,
params: unknown[] = [],
): Promise<Record<string, unknown>[]> {
const primary = this.pread(query, params);
primary.catch(() => {});
if (!this.hasReadReplica()) return this.read(query, params);
const primary = settle(this.pread(query, params));
try {
const rows = await this.read(query, params);
if (rows.length > 0) {
return rows;
}
const rows = await this.replicaRead(query, params);
if (rows.length > 0) return rows;
} catch {
// replica failed — fall through to primary
}
return primary;
const result = await primary;
if ('error' in result) throw result.error;
return result.value;
}
/** Like `tryHardRead()` but throws when the result set is empty. */
async requireRead(
query: string,
params: unknown[] = [],
): Promise<Record<string, unknown>[]> {
const rows = await this.tryHardRead(query, params);
if (rows.length === 0) {
throw new Error(`required read returned no rows: ${query}`);
/**
* Apply the `.sql` files named `<engineName>*` in each configured migration
* directory, in filename order. Files must be idempotent: nothing records
* which ones already ran. A failure aborts startup. `transactional` wraps
* each file in BEGIN/COMMIT, for engines whose DDL can roll back.
*/
protected async applyMigrationPaths(
connect: () => Promise<MigrationConnection>,
splitStatements: (contents: string) => string[],
{ transactional }: { transactional: boolean },
): Promise<void> {
const paths = this.config.database?.migrationPaths;
if (!paths || paths.length === 0) return;
const tag = `[${this.engineName}]`;
const conn = await connect();
try {
for (const rawPath of paths) {
const dir = isAbsolute(rawPath)
? rawPath
: resolvePath(process.cwd(), rawPath);
let files: string[];
try {
files = readdirSync(dir)
.filter(
(f) =>
f.endsWith('.sql') &&
f.startsWith(this.engineName),
)
.sort(compareMigrationFilenames);
} catch (e) {
throw new Error(
`${tag} migration path is unreadable: ${dir}`,
{
cause: e,
},
);
}
if (files.length === 0) {
console.log(`${tag} no migrations in ${dir}`);
continue;
}
console.log(
`${tag} running migrations from ${dir}: ${files.length} file(s)`,
);
for (const file of files) {
const contents = readFileSync(
resolvePath(dir, file),
'utf8',
);
const statements = splitStatements(contents);
if (transactional) await conn.query('BEGIN');
try {
for (let i = 0; i < statements.length; i++) {
try {
await conn.query(statements[i]);
} catch (e) {
throw new Error(
`${tag} failed to apply ${file} at statement ${i}`,
{ cause: e },
);
}
}
if (transactional) await conn.query('COMMIT');
} catch (e) {
if (transactional) {
try {
await conn.query('ROLLBACK');
} catch {
// the original failure is the one worth surfacing
}
}
throw e;
}
console.log(
`${tag} applied ${file} (${statements.length} statements)`,
);
}
}
} finally {
conn.release();
}
return rows;
}
/**
@@ -226,6 +336,30 @@ export class AbstractDatabaseClient extends PuterClient {
});
}
/**
* Whole `periodSeconds` buckets since the epoch, from a unix-seconds
* column. MySQL's `/` yields a decimal, which would split a bucket.
*/
periodBucket(secondsExpression: string, periodSeconds: number): string {
return this.case({
mysql: `(${secondsExpression} DIV ${periodSeconds})`,
otherwise: `(${secondsExpression} / ${periodSeconds})`,
});
}
/**
* Counts each (`userExpression`, bucket) pair once. `COUNT(DISTINCT a, b)`
* is MySQL-only, so the pair folds into one number — injective while the
* bucket stays under the multiplier, which hourly reaches in the 2980s.
*/
countDistinctUserPeriods(
userExpression: string,
secondsExpression: string,
periodSeconds: number,
): string {
return `COUNT(DISTINCT ${userExpression} * 10000000 + ${this.periodBucket(secondsExpression, periodSeconds)})`;
}
nullCoalesce(...expressions: readonly string[]): string {
if (expressions.length === 0) {
throw new Error('nullCoalesce requires at least one expression');
@@ -240,7 +374,43 @@ export class AbstractDatabaseClient extends PuterClient {
});
}
/**
* A timestamp column value from one bound parameter in unix seconds, stored
* the way `CURRENT_TIMESTAMP` would store that instant.
*/
timestampFromUnixSql(): string {
return this.case({
sqlite: "datetime(?, 'unixepoch')",
postgres: 'to_timestamp(?)',
otherwise: 'FROM_UNIXTIME(?)',
});
}
/**
* What a read of a timestamp column holding `seconds` returns: sqlite gives
* back the UTC text it stored, mysql2 and pg a `Date` read in node's time
* zone (so this matches when node and the database session share one).
*/
timestampValue(seconds: number): Date | string {
const date = new Date(seconds * 1000);
return this.case<Date | string>({
sqlite: date.toISOString().slice(0, 19).replace('T', ' '),
otherwise: date,
});
}
protected sqlStringLiteral(value: string): string {
return `'${value.replaceAll("'", "''")}'`;
}
}
/** Await later without an unhandled rejection in the meantime. */
const settle = async <T>(
promise: Promise<T>,
): Promise<{ value: T } | { error: unknown }> => {
try {
return { value: await promise };
} catch (error) {
return { error };
}
};
@@ -184,7 +184,8 @@ describe('compareMigrationFilenames', () => {
//
// `read()` normally goes to the replica batcher; when the replica side is
// degraded (batcher load-shed or a transient connection error) and a real
// replica is configured, the read retries once on the primary batcher.
// replica is configured, the read retries once on the primary read batcher.
// The write batcher is left unset, so a failover that reached it would throw.
type Batcher = { execute: ReturnType<typeof vi.fn> };
@@ -200,7 +201,7 @@ const makeClient = (opts: {
// inject the batchers directly. Configuration enum: SINGLE=0, REPLICA=1.
Object.assign(client as unknown as Record<string, unknown>, {
dbReplica: opts.replica,
db: opts.primary,
dbPrimaryRead: opts.primary,
configuration: opts.multiNode === false ? 0 : 1,
});
return client;
@@ -284,6 +285,10 @@ const startClient = async (
// must answer with one row-set per statement plus one for the SELECT 1.
const rowSets = (...sets: unknown[][]) => [[...sets, [{ 1: 1 }]], undefined];
// Writes go out one statement at a time, so the driver answers with just the
// result header.
const writeHeader = (header: Record<string, unknown>) => [header, undefined];
beforeEach(() => {
createdPools.length = 0;
createPoolMock.mockClear();
@@ -303,6 +308,9 @@ describe('MySQLDatabaseClient — pool construction', () => {
expect(createdPools[0].poolConfig).toEqual({
maxPreparedStatements: 900,
connectionLimit: 30,
maxIdle: 29,
idleTimeout: 60_000,
gracefulEnd: true,
enableKeepAlive: true,
host: 'db.internal',
port: 3307,
@@ -313,6 +321,32 @@ describe('MySQLDatabaseClient — pool construction', () => {
});
});
// mysql2 skips its idle reaper unless maxIdle < connectionLimit, and an
// unreaped connection is closed by the server under the pool's feet.
it('keeps the idle reaper on when a pool sets its own connection limit', async () => {
await startClient({
replica: { host: 'replica.internal', connectionLimit: 10 },
});
expect(createdPools[1].poolConfig).toMatchObject({
connectionLimit: 10,
maxIdle: 9,
idleTimeout: 60_000,
});
});
// mysql2 reads 0 as unlimited; a maxIdle of -1 would crash its reaper.
it('leaves maxIdle to mysql2 for an unlimited pool', async () => {
await startClient({
replica: { host: 'replica.internal', connectionLimit: 0 },
});
expect(createdPools[1].poolConfig).toMatchObject({
connectionLimit: 0,
});
expect(createdPools[1].poolConfig).not.toHaveProperty('maxIdle');
});
it('falls back to loopback defaults when the endpoint is unspecified', async () => {
await startClient();
@@ -401,10 +435,129 @@ describe('MySQLDatabaseClient — query interface', () => {
]);
});
it.each([
{ method: 'pread', poolIndex: 0, maxTimeInQueue: 30 },
{ method: 'read', poolIndex: 1, maxTimeInQueue: 10 },
] as const)(
'$method: flushes lone reads at once until 8 are in flight, then waits $maxTimeInQueue ms and coalesces',
async ({ method, poolIndex, maxTimeInQueue }) => {
const client = await startClient({ replica: { host: 'replica' } });
const pool = createdPools[poolIndex];
const held: Array<() => void> = [];
pool.respond = (sql) =>
new Promise((resolve) => {
const sets = sql
.split(';')
.slice(0, -1)
.map((stmt) => [{ stmt }]);
held.push(() => resolve(rowSets(...sets)));
});
vi.useFakeTimers({
toFake: [
'setTimeout',
'clearTimeout',
'setImmediate',
'clearImmediate',
],
});
try {
const reads: Promise<unknown>[] = [];
for (let i = 0; i < 8; i++) {
reads.push(client[method](`SELECT ${i}`));
await vi.advanceTimersByTimeAsync(0);
}
expect(pool.calls).toHaveLength(8);
reads.push(client[method]('SELECT 8'));
await vi.advanceTimersByTimeAsync(maxTimeInQueue - 1);
reads.push(client[method]('SELECT 9'));
expect(pool.calls).toHaveLength(8);
await vi.advanceTimersByTimeAsync(1);
expect(pool.calls.map((c) => c.sql).at(-1)).toBe(
'SELECT 8;SELECT 9; SELECT 1',
);
for (const release of held.splice(0)) release();
const rows = await Promise.all(reads);
expect(rows.at(-2)).toEqual([{ stmt: 'SELECT 8' }]);
expect(rows.at(-1)).toEqual([{ stmt: 'SELECT 9' }]);
} finally {
vi.useRealTimers();
}
},
);
it.each([
{ setup: 'single node', replica: undefined, waits: true },
{ setup: 'replica', replica: { host: 'replica' }, waits: false },
])(
'$setup: 8 primary reads in flight hold back a lone read: $waits',
async ({ replica, waits }) => {
const client = await startClient(replica ? { replica } : {});
const held: Array<() => void> = [];
for (const pool of createdPools) {
pool.respond = (sql) =>
new Promise((resolve) => {
const sets = sql
.split(';')
.slice(0, -1)
.map((stmt) => [{ stmt }]);
held.push(() => resolve(rowSets(...sets)));
});
}
const sent = () => createdPools.flatMap((pool) => pool.calls);
vi.useFakeTimers({
toFake: [
'setTimeout',
'clearTimeout',
'setImmediate',
'clearImmediate',
],
});
try {
const reads: Promise<unknown>[] = [];
for (let i = 0; i < 8; i++) {
reads.push(client.pread(`SELECT ${i}`));
await vi.advanceTimersByTimeAsync(0);
}
reads.push(client.read('SELECT 8'));
await vi.advanceTimersByTimeAsync(0);
expect(sent()).toHaveLength(waits ? 8 : 9);
await vi.advanceTimersByTimeAsync(10);
expect(sent()).toHaveLength(9);
for (const release of held.splice(0)) release();
await Promise.all(reads);
} finally {
vi.useRealTimers();
}
},
);
it('sends a write on its own, outside a transaction', async () => {
const client = await startClient();
const pool = createdPools[0];
pool.respond = () => writeHeader({ affectedRows: 1 });
await client.write('UPDATE `user` SET `username` = ? WHERE `id` = ?', [
'ada',
1,
]);
expect(pool.calls).toEqual([
{
sql: 'UPDATE `user` SET `username` = ? WHERE `id` = ?',
values: ['ada', 1],
},
]);
expect(pool.connections[0].beginTransaction).not.toHaveBeenCalled();
expect(pool.connections[0].commit).not.toHaveBeenCalled();
});
it('maps the mysql result header onto the shared write result', async () => {
const client = await startClient();
createdPools[0].respond = () =>
rowSets({ insertId: 42, affectedRows: 2 } as unknown as unknown[]);
writeHeader({ insertId: 42, affectedRows: 2 });
await expect(
client.write('UPDATE `user` SET `username` = ? WHERE `id` = ?', [
@@ -418,9 +571,26 @@ describe('MySQLDatabaseClient — query interface', () => {
});
});
it('passes the warning count through, so callers can tell a clean write', async () => {
const client = await startClient();
createdPools[0].respond = () =>
writeHeader({ insertId: 9, affectedRows: 1, warningStatus: 1 });
await expect(
client.write('INSERT IGNORE INTO `user` (`username`) VALUES (?)', [
'ada',
]),
).resolves.toEqual({
insertId: 9,
affectedRows: 1,
anyRowsAffected: true,
warningStatus: 1,
});
});
it('reports no rows affected when the header omits the counters', async () => {
const client = await startClient();
createdPools[0].respond = () => rowSets({} as unknown as unknown[]);
createdPools[0].respond = () => writeHeader({});
await expect(client.write('DELETE FROM `user`')).resolves.toEqual({
insertId: 0,
@@ -432,17 +602,14 @@ describe('MySQLDatabaseClient — query interface', () => {
it('builds an INSERT with quoted identifiers and positional params', async () => {
const client = await startClient();
const pool = createdPools[0];
pool.respond = () =>
rowSets({ insertId: 7, affectedRows: 1 } as unknown as unknown[]);
pool.respond = () => writeHeader({ insertId: 7, affectedRows: 1 });
await expect(
client.insert('user', { username: 'ada', email: null }),
).resolves.toMatchObject({ insertId: 7 });
expect(pool.calls[0]).toEqual({
sql:
'INSERT INTO `user` (`username`, `email`) VALUES (?, ?); ' +
'SELECT 1',
sql: 'INSERT INTO `user` (`username`, `email`) VALUES (?, ?)',
values: ['ada', null],
});
});
@@ -568,6 +735,21 @@ describe('MySQLDatabaseClient — tryHardRead', () => {
{ from: 'primary' },
]);
});
it('reads the primary once when the replica fails in a way read() fails over on', async () => {
const replica = {
execute: vi.fn().mockRejectedValue(codedError('dbBatchFailed')),
};
const primary = {
execute: vi.fn().mockResolvedValue([[{ from: 'primary' }]]),
};
const client = makeClient({ replica, primary });
await expect(client.tryHardRead('SELECT 1')).resolves.toEqual([
{ from: 'primary' },
]);
expect(primary.execute).toHaveBeenCalledTimes(1);
});
});
describe('MySQLDatabaseClient — migrations', () => {
@@ -645,46 +827,6 @@ describe('MySQLDatabaseClient — migrations', () => {
});
describe('MySQLDatabaseClient — pool lifecycle', () => {
it('replaces and closes the previous primary pool on reinit', async () => {
const client = await startClient();
const original = createdPools[0];
client.reinitPrimary();
expect(createdPools).toHaveLength(2);
expect(original.ended).toBe(true);
// Single-node: the replica batcher must follow the new primary.
createdPools[1].respond = () => rowSets([{ from: 'new-primary' }]);
await expect(client.read('SELECT 1')).resolves.toEqual([
{ from: 'new-primary' },
]);
});
it('replaces only the replica pool on replica reinit', async () => {
const client = await startClient({ replica: { host: 'replica' } });
const [primary, originalReplica] = createdPools;
client.reinitReplica();
expect(createdPools).toHaveLength(3);
expect(originalReplica.ended).toBe(true);
expect(primary.ended).toBe(false);
});
it('ignores a replica reinit when no replica is configured', async () => {
const client = await startClient();
client.reinitReplica();
expect(createdPools).toHaveLength(1);
});
it('ignores reinit once shutdown has started', async () => {
const client = await startClient();
await client.onServerPrepareShutdown();
client.reinitPrimary();
expect(createdPools).toHaveLength(1);
await client.onServerShutdown();
});
it('closes both pools on shutdown when a replica is configured', async () => {
const client = await startClient({ replica: { host: 'replica' } });
await client.onServerShutdown();
@@ -728,108 +870,3 @@ describe('MySQLDatabaseClient — pool lifecycle', () => {
await expect(client.onServerShutdown()).rejects.toThrow('end exploded');
});
});
describe('MySQLDatabaseClient.readWithRetry', () => {
const codedFailure = (code: string) => {
const err = new Error(code) as Error & { code: string };
err.code = code;
return err;
};
it('classifies transient connection failures as retriable', () => {
expect(
MySQLDatabaseClient.isRetriableError(codedFailure('ECONNRESET')),
).toBe(true);
expect(
MySQLDatabaseClient.isRetriableError(codedFailure('ER_DUP_ENTRY')),
).toBe(false);
expect(
MySQLDatabaseClient.isRetriableError(new Error('Connection lost')),
).toBe(true);
});
it('retries a transient failure and returns the eventual result', async () => {
const client = await startClient();
let attempts = 0;
const operation = vi.fn(async () => {
attempts += 1;
if (attempts < 3) throw codedFailure('PROTOCOL_CONNECTION_LOST');
return [{ ok: 1 }];
});
await expect(
client.readWithRetry('health', operation, { baseBackoffMs: 0 }),
).resolves.toEqual([{ ok: 1 }]);
expect(operation).toHaveBeenCalledTimes(3);
});
it('gives up after the attempt budget is spent', async () => {
const client = await startClient();
const operation = vi.fn(async () => {
throw codedFailure('ETIMEDOUT');
});
await expect(
client.readWithRetry('health', operation, {
maxAttempts: 2,
baseBackoffMs: 0,
}),
).rejects.toMatchObject({ code: 'ETIMEDOUT' });
expect(operation).toHaveBeenCalledTimes(2);
});
it('never retries a deterministic SQL error', async () => {
const client = await startClient();
const operation = vi.fn(async () => {
throw codedFailure('ER_PARSE_ERROR');
});
await expect(
client.readWithRetry('health', operation, { baseBackoffMs: 0 }),
).rejects.toMatchObject({ code: 'ER_PARSE_ERROR' });
expect(operation).toHaveBeenCalledTimes(1);
});
it('stops retrying once shutdown has started', async () => {
const client = await startClient();
await client.onServerPrepareShutdown();
const operation = vi.fn(async () => {
throw codedFailure('ECONNRESET');
});
await expect(
client.readWithRetry('health', operation, { baseBackoffMs: 0 }),
).rejects.toMatchObject({ code: 'ECONNRESET' });
expect(operation).toHaveBeenCalledTimes(1);
await client.onServerShutdown();
});
it('applies jittered exponential backoff capped at maxBackoffMs', async () => {
const client = await startClient();
const delays: number[] = [];
const realSetTimeout = globalThis.setTimeout;
const timeoutSpy = vi
.spyOn(globalThis, 'setTimeout')
.mockImplementation(((fn: () => void, ms?: number) => {
delays.push(ms ?? 0);
return realSetTimeout(fn, 0);
}) as typeof setTimeout);
try {
const operation = vi.fn(async () => {
throw codedFailure('ECONNRESET');
});
await expect(
client.readWithRetry('health', operation, {
maxAttempts: 4,
baseBackoffMs: 100,
maxBackoffMs: 150,
jitterRatio: 0,
}),
).rejects.toMatchObject({ code: 'ECONNRESET' });
expect(delays).toEqual([100, 150, 150]);
} finally {
timeoutSpy.mockRestore();
}
});
});
@@ -17,8 +17,6 @@
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { readdirSync, readFileSync } from 'fs';
import { isAbsolute, resolve as resolvePath } from 'path';
import { metrics } from '@opentelemetry/api';
import { createPool, ExecuteValues, type Pool } from 'mysql2';
import { Span } from '../../util/span.js';
@@ -26,10 +24,11 @@ import { AbstractDatabaseClient, type WriteResult } from './DatabaseClient';
import { SQLBatcher } from './SQLBatcher.js';
import { isRetriableError } from './retriableErrors.js';
import { splitMysqlStatements } from './splitMysqlStatements.js';
import { compareMigrationFilenames } from './migrationFilenames.js';
import type { IConfig } from '../../types';
const DEFAULT_SELECT_TIMEOUT_MS = 30_000;
const DEFAULT_CONNECTION_LIMIT = 30;
const IDLE_TIMEOUT_MS = 60_000;
const replicaFailoverCounter = metrics
.getMeter('puter-backend')
@@ -38,7 +37,7 @@ const replicaFailoverCounter = metrics
'Reads that failed on the replica batcher and were retried on the primary',
});
export { compareMigrationFilenames };
export { compareMigrationFilenames } from './migrationFilenames.js';
type PoolConfig = Parameters<typeof createPool>[0];
@@ -54,7 +53,7 @@ export class MySQLDatabaseClient extends AbstractDatabaseClient {
private replicaPool!: Pool;
private db!: SQLBatcher;
private dbReplica!: SQLBatcher;
/** Primary pool, SELECT-only: same rows as `db`, without the transaction. */
/** Primary pool, SELECT-only: same rows as `db`, coalesced like the replica. */
private dbPrimaryRead!: SQLBatcher;
private configuration = Configuration.SINGLE;
private shutdownStarted = false;
@@ -79,8 +78,13 @@ export class MySQLDatabaseClient extends AbstractDatabaseClient {
});
console.log('[mysql] connected to primary');
// Read batchers on one pool share the cap on immediate flushes.
const primaryReadFlushes = { count: 0 };
this.db = this.createPrimaryBatcher(this.primaryPool);
this.dbPrimaryRead = this.createPrimaryReadBatcher(this.primaryPool);
this.dbPrimaryRead = this.createPrimaryReadBatcher(
this.primaryPool,
primaryReadFlushes,
);
if (dbConf.replica) {
this.replicaPool = this.createPool(dbConf.replica);
@@ -91,17 +95,26 @@ export class MySQLDatabaseClient extends AbstractDatabaseClient {
this.configuration = Configuration.SINGLE;
}
this.dbReplica = this.createReplicaBatcher(this.replicaPool);
this.dbReplica = this.createReplicaBatcher(
this.replicaPool,
this.configuration === Configuration.SINGLE
? primaryReadFlushes
: { count: 0 },
);
await this.runMigrations();
await this.applyMigrationPaths(
() => this.primaryPool.promise().getConnection(),
splitMysqlStatements,
{ transactional: false },
);
}
override async onServerPrepareShutdown(): Promise<void> {
if (this.shutdownStarted) return;
this.shutdownStarted = true;
// Blocks reinitPrimary/reinitReplica from here on. Pools stay open;
// onServerShutdown closes them after the layers above have drained.
// Pools stay open; onServerShutdown closes them after the layers
// above have drained.
console.log('[mysql] entering drain mode');
}
@@ -136,7 +149,7 @@ export class MySQLDatabaseClient extends AbstractDatabaseClient {
throw error;
}
replicaFailoverCounter.add(1);
result = await this.db.execute(query, params);
result = await this.dbPrimaryRead.execute(query, params);
}
if (!result) return [];
return (result[0] as Record<string, unknown>[]) ?? [];
@@ -161,12 +174,14 @@ export class MySQLDatabaseClient extends AbstractDatabaseClient {
const header = result[0] as {
insertId?: number;
affectedRows?: number;
warningStatus?: number;
};
const affectedRows = header.affectedRows ?? 0;
return {
insertId: header.insertId ?? 0,
affectedRows,
anyRowsAffected: affectedRows > 0,
warningStatus: header.warningStatus,
};
}
@@ -198,101 +213,18 @@ export class MySQLDatabaseClient extends AbstractDatabaseClient {
}
}
@Span('db.tryHardRead', (query: string) => ({ 'db.statement': query }))
override async tryHardRead(
query: string,
params: unknown[] = [],
): Promise<Record<string, unknown>[]> {
if (this.configuration === Configuration.SINGLE) {
return this.read(query, params);
}
// Run both reads in parallel — prefer replica when it returns rows,
// otherwise fall back to primary to handle replication lag.
const primaryPromise = this.dbPrimaryRead.execute(query, params);
try {
const replicaResult = await this.dbReplica.execute(query, params);
if (
Array.isArray(replicaResult?.[0]) &&
(replicaResult[0] as unknown[]).length > 0
) {
primaryPromise.catch(() => {}); // suppress unhandled rejection
return replicaResult[0] as Record<string, unknown>[];
}
} catch {
// fall through to primary
}
const primaryResult = await primaryPromise;
return (primaryResult?.[0] as Record<string, unknown>[]) ?? [];
protected override hasReadReplica(): boolean {
return this.configuration === Configuration.REPLICA;
}
// ------------------------------------------------------------------
// Migrations
// ------------------------------------------------------------------
/**
* Apply `.sql` files from each configured migration directory in order.
* Files within a directory are sorted lexically. Files MUST be idempotent —
* there is no per-file applied-state tracking. Failures abort startup so
* operators see schema problems loud.
*/
private async runMigrations(): Promise<void> {
const paths = this.config.database?.migrationPaths;
if (!paths || paths.length === 0) return;
const conn = await this.primaryPool.promise().getConnection();
try {
for (const rawPath of paths) {
const dir = isAbsolute(rawPath)
? rawPath
: resolvePath(process.cwd(), rawPath);
let files: string[];
try {
files = readdirSync(dir)
.filter(
(f) => f.endsWith('.sql') && f.startsWith('mysql'),
)
.sort(compareMigrationFilenames);
} catch (e) {
throw new Error(
`[mysql] migration path is unreadable: ${dir}`,
{ cause: e },
);
}
if (files.length === 0) {
console.log(`[mysql] no migrations in ${dir}`);
continue;
}
console.log(
`[mysql] running migrations from ${dir}: ${files.length} file(s)`,
);
for (const file of files) {
const filePath = resolvePath(dir, file);
const contents = readFileSync(filePath, 'utf8');
const statements = splitMysqlStatements(contents);
for (let i = 0; i < statements.length; i++) {
try {
await conn.query(statements[i]);
} catch (e) {
throw new Error(
`[mysql] failed to apply ${file} at statement ${i}`,
{ cause: e },
);
}
}
console.log(
`[mysql] applied ${file} (${statements.length} statements)`,
);
}
}
} finally {
conn.release();
}
// No failover: `tryHardRead` already has a primary read in flight.
protected override async replicaRead(
query: string,
params: unknown[],
): Promise<Record<string, unknown>[]> {
const result = await this.dbReplica.execute(query, params);
if (!result) return [];
return (result[0] as Record<string, unknown>[]) ?? [];
}
// ------------------------------------------------------------------
@@ -300,9 +232,19 @@ export class MySQLDatabaseClient extends AbstractDatabaseClient {
// ------------------------------------------------------------------
private createPool(poolConfig: PoolConfig): Pool {
const connectionLimit =
poolConfig.connectionLimit ?? DEFAULT_CONNECTION_LIMIT;
const pool = createPool({
maxPreparedStatements: 900,
connectionLimit: 30,
connectionLimit,
// mysql2 only reaps idle connections when maxIdle < connectionLimit.
// Unreaped, they sit until the server's wait_timeout closes them,
// and the next statement sent on one fails. 0 means unlimited,
// and a negative maxIdle would crash the reaper.
...(connectionLimit > 1 && { maxIdle: connectionLimit - 1 }),
idleTimeout: IDLE_TIMEOUT_MS,
// Reaped connections say goodbye instead of dropping the socket.
gracefulEnd: true,
enableKeepAlive: true,
...poolConfig,
multipleStatements: true,
@@ -330,24 +272,28 @@ export class MySQLDatabaseClient extends AbstractDatabaseClient {
return pool;
}
/**
* One write per flush. Coalesced writes share a transaction, so unrelated
* callers hold each other's row locks until COMMIT and deadlock; a lone
* statement needs no BEGIN/COMMIT round trips.
*/
private createPrimaryBatcher(pool: Pool): SQLBatcher {
return new SQLBatcher(pool, {
maxTimeInQueue: 30,
maxBatchSize: 5,
maxBatchSize: 1,
poolLabel: 'primary',
acquireTimeoutMs: this.config.database?.acquireTimeoutMs,
});
}
/**
* Reads that must see the primary still only read, so they skip the
* batcher's transaction wrapper. Worth a separate batcher because BEGIN and
* COMMIT are round trips: against a primary in another region they cost
* more than the query.
*/
private createPrimaryReadBatcher(pool: Pool): SQLBatcher {
/** Reads that must see the primary, still coalesced. */
private createPrimaryReadBatcher(
pool: Pool,
inFlight: { count: number },
): SQLBatcher {
return new SQLBatcher(pool, {
maxTimeInQueue: 30,
maxInFlightImmediate: 8,
inFlight,
maxBatchSize: 5,
poolLabel: 'primary',
readOnly: true,
@@ -355,9 +301,14 @@ export class MySQLDatabaseClient extends AbstractDatabaseClient {
});
}
private createReplicaBatcher(pool: Pool): SQLBatcher {
private createReplicaBatcher(
pool: Pool,
inFlight: { count: number },
): SQLBatcher {
return new SQLBatcher(pool, {
maxTimeInQueue: 10,
maxInFlightImmediate: 8,
inFlight,
maxBatchSize: 5,
poolLabel: 'replica',
readOnly: true,
@@ -365,57 +316,6 @@ export class MySQLDatabaseClient extends AbstractDatabaseClient {
});
}
/** Reinitialize the primary pool (e.g. after a health-check failure). */
reinitPrimary(): void {
if (this.shutdownStarted) return;
const dbConf = this.config.database!;
const previous = this.primaryPool;
this.primaryPool = this.createPool({
host: dbConf.host ?? '127.0.0.1',
port: dbConf.port ?? 3306,
user: dbConf.user ?? 'root',
password: dbConf.password ?? '',
database: dbConf.database ?? 'puter',
});
this.db = this.createPrimaryBatcher(this.primaryPool);
this.dbPrimaryRead = this.createPrimaryReadBatcher(this.primaryPool);
if (this.configuration === Configuration.SINGLE) {
this.replicaPool = this.primaryPool;
this.dbReplica = this.createReplicaBatcher(this.primaryPool);
}
if (previous && previous !== this.primaryPool) {
this.closePool(previous, 'reinit:primary').catch(() => {});
}
}
/** Reinitialize the replica pool. */
reinitReplica(): void {
if (this.shutdownStarted || !this.config.database?.replica) return;
const previous = this.replicaPool;
this.replicaPool = this.createPool(this.config.database.replica);
this.dbReplica = this.createReplicaBatcher(this.replicaPool);
if (
previous &&
previous !== this.replicaPool &&
previous !== this.primaryPool
) {
this.closePool(previous, 'reinit:replica').catch(() => {});
}
}
// ------------------------------------------------------------------
// Retry helpers (for health checks or resilient reads)
// ------------------------------------------------------------------
static isRetriableError(error: unknown): boolean {
return isRetriableError(error);
}
/**
* Replica failures worth retrying on the primary: batcher load-shed or
* transient connection errors — never deterministic SQL errors.
@@ -425,52 +325,6 @@ export class MySQLDatabaseClient extends AbstractDatabaseClient {
return code === 'dbBatchFailed' || isRetriableError(error);
}
async readWithRetry(
label: string,
operation: () => Promise<unknown[]>,
opts?: {
maxAttempts?: number;
baseBackoffMs?: number;
maxBackoffMs?: number;
jitterRatio?: number;
},
): Promise<unknown[]> {
const maxAttempts = opts?.maxAttempts ?? 3;
const baseBackoffMs = opts?.baseBackoffMs ?? 100;
const maxBackoffMs = opts?.maxBackoffMs ?? 500;
const jitterRatio = opts?.jitterRatio ?? 0.2;
let attempt = 1;
while (true) {
try {
return await operation();
} catch (error) {
if (this.shutdownStarted) throw error;
if (
attempt >= maxAttempts ||
!MySQLDatabaseClient.isRetriableError(error)
)
throw error;
const raw = baseBackoffMs * 2 ** (attempt - 1);
const capped = Math.min(maxBackoffMs, raw);
const window = Math.round(capped * jitterRatio);
const jitter =
window === 0
? 0
: Math.floor(Math.random() * (window * 2 + 1)) - window;
const delay = Math.max(0, capped + jitter);
console.warn(
`[${label}] transient mysql error (${(error as { code?: string })?.code ?? 'unknown'}); retry ${attempt + 1}/${maxAttempts} in ${delay}ms`,
);
await new Promise((r) => setTimeout(r, delay));
attempt++;
}
}
}
// ------------------------------------------------------------------
// Internal pool lifecycle
// ------------------------------------------------------------------
@@ -28,6 +28,20 @@ import {
type PostgresPoolClient,
} from './PostgresDatabaseClient.js';
// Pass-through spy: the rewrite is pure, so its call count shows the memo.
const { rewriteSpy } = vi.hoisted(() => ({ rewriteSpy: vi.fn() }));
vi.mock('./preparePostgresSql.js', async (importOriginal) => {
const actual =
await importOriginal<typeof import('./preparePostgresSql.js')>();
return {
...actual,
preparePostgresSql: (sql: string) => {
rewriteSpy(sql);
return actual.preparePostgresSql(sql);
},
};
});
type QueryCall = {
text: string;
values?: unknown[];
@@ -167,6 +181,23 @@ describe('PostgresDatabaseClient', () => {
});
});
it('rewrites a repeated query once', async () => {
const pool = new RecordingPool();
const client = new PostgresDatabaseClient(postgresConfig(), () => pool);
await client.onServerStart();
const query = 'SELECT `name` FROM `apps` WHERE `id` = ?';
rewriteSpy.mockClear();
await client.read(query, [1]);
await client.pread(query, [2]);
await client.write(query, [3]);
expect(rewriteSpy).toHaveBeenCalledTimes(1);
expect(pool.calls.slice(-3).map((c) => c.text)).toEqual(
Array(3).fill('SELECT "name" FROM "apps" WHERE "id" = $1'),
);
});
it('normalizes int8 fields on read and primary read rows', async () => {
const pool = new RecordingPool(
new RecordingPoolClient(),
@@ -17,16 +17,14 @@
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { readdirSync, readFileSync } from 'fs';
import { isAbsolute, resolve as resolvePath } from 'path';
import { Pool, type PoolConfig, type QueryResult } from 'pg';
import { BoundedTtlMap } from '../../util/boundedTtlMap.js';
import { Span } from '../../util/span.js';
import {
AbstractDatabaseClient,
type BatchEntry,
type WriteResult,
} from './DatabaseClient';
import { compareMigrationFilenames } from './migrationFilenames.js';
import { preparePostgresSql } from './preparePostgresSql.js';
import { splitPostgresStatements } from './splitPostgresStatements.js';
import type { IConfig } from '../../types';
@@ -62,6 +60,7 @@ enum Configuration {
}
const POSTGRES_INT8_OID = 20;
const PREPARED_SQL_CACHE_SIZE = 1000;
const INTEGER_TEXT_PATTERN = /^-?\d+$/u;
const normalizePostgresInt8 = (
@@ -138,6 +137,10 @@ export class PostgresDatabaseClient extends AbstractDatabaseClient {
private replicaPool!: PostgresPool;
private configuration = Configuration.SINGLE;
private shutdownStarted = false;
/** Query text as written → its Postgres rewrite, which is pure. */
private readonly preparedSql = new BoundedTtlMap<string, string>({
maxEntries: PREPARED_SQL_CACHE_SIZE,
});
constructor(
config: IConfig,
@@ -164,7 +167,11 @@ export class PostgresDatabaseClient extends AbstractDatabaseClient {
this.configuration = Configuration.SINGLE;
}
await this.runMigrations();
await this.applyMigrationPaths(
() => this.primaryPool.connect(),
splitPostgresStatements,
{ transactional: true },
);
}
override async onServerPrepareShutdown(): Promise<void> {
@@ -248,98 +255,8 @@ export class PostgresDatabaseClient extends AbstractDatabaseClient {
}
}
@Span('db.tryHardRead', (query: string) => ({ 'db.statement': query }))
override async tryHardRead(
query: string,
params: unknown[] = [],
): Promise<Record<string, unknown>[]> {
if (this.configuration === Configuration.SINGLE) {
return this.read(query, params);
}
const primaryPromise = this.query(this.primaryPool, query, params);
try {
const replicaResult = await this.query(
this.replicaPool,
query,
params,
);
if (replicaResult.rows.length > 0) {
primaryPromise.catch(() => {});
return normalizePostgresRows(replicaResult);
}
} catch {
// fall through to primary
}
const primaryResult = await primaryPromise;
return normalizePostgresRows(primaryResult);
}
private async runMigrations(): Promise<void> {
const paths = this.config.database?.migrationPaths;
if (!paths || paths.length === 0) return;
const conn = await this.primaryPool.connect();
try {
for (const rawPath of paths) {
const dir = isAbsolute(rawPath)
? rawPath
: resolvePath(process.cwd(), rawPath);
let files: string[];
try {
files = readdirSync(dir)
.filter(
(f) =>
f.endsWith('.sql') && f.startsWith('postgres'),
)
.sort(compareMigrationFilenames);
} catch (e) {
throw new Error(
`[postgres] migration path is unreadable: ${dir}`,
{ cause: e },
);
}
if (files.length === 0) {
console.log(`[postgres] no migrations in ${dir}`);
continue;
}
console.log(
`[postgres] running migrations from ${dir}: ${files.length} file(s)`,
);
for (const file of files) {
const filePath = resolvePath(dir, file);
const contents = readFileSync(filePath, 'utf8');
const statements = splitPostgresStatements(contents);
await conn.query('BEGIN');
try {
for (let i = 0; i < statements.length; i++) {
try {
await conn.query(statements[i]);
} catch (e) {
throw new Error(
`[postgres] failed to apply ${file} at statement ${i}`,
{ cause: e },
);
}
}
await conn.query('COMMIT');
} catch (e) {
await conn.query('ROLLBACK').catch(() => {});
throw e;
}
console.log(
`[postgres] applied ${file} (${statements.length} statements)`,
);
}
}
} finally {
conn.release();
}
protected override hasReadReplica(): boolean {
return this.configuration === Configuration.REPLICA;
}
private createPool(dbConf: PostgresEndpointConfig): PostgresPool {
@@ -366,8 +283,12 @@ export class PostgresDatabaseClient extends AbstractDatabaseClient {
query: string,
params: unknown[] = [],
): Promise<QueryResult> {
const prepared = preparePostgresSql(query);
return target.query(prepared.text, params);
let text = this.preparedSql.get(query);
if (text === undefined) {
text = preparePostgresSql(query).text;
this.preparedSql.set(query, text);
}
return target.query(text, params);
}
private async closeCurrentPools(): Promise<void> {
+103 -34
View File
@@ -76,6 +76,7 @@ const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
export class SQLBatcher {
dbPool;
maxTimeInQueue;
maxInFlightImmediate;
maxBatchSize;
maxQueueSize;
failureThreshold;
@@ -84,7 +85,9 @@ export class SQLBatcher {
readOnly;
acquireTimeoutMs;
queue = [];
timeouts = [];
#flushTimer = null;
#flushTimerIsImmediate = false;
#inFlight;
#consecutiveFailures = 0;
#lastFailureAt = 0;
#metricAttrs;
@@ -93,6 +96,11 @@ export class SQLBatcher {
* @param {object} dbPool Mysql2 pool
* @param {object} [opts]
* @param {number} [opts.maxTimeInQueue] Ms an item may wait before flush
* @param {number} [opts.maxInFlightImmediate] While fewer flushes than this
* are in flight, a lone item flushes on the next turn instead of waiting
* maxTimeInQueue; past it, items wait and coalesce to spare connections
* @param {{ count: number }} [opts.inFlight] In-flight flush counter the
* cap reads; batchers sharing a pool can share one
* @param {number} [opts.maxBatchSize] Items coalesced per flush
* @param {number} [opts.maxQueueSize] Drop-oldest high-water mark
* @param {number} [opts.failureThreshold] Consecutive failures to open the
@@ -111,6 +119,8 @@ export class SQLBatcher {
dbPool,
{
maxTimeInQueue = 20,
maxInFlightImmediate = 0,
inFlight = { count: 0 },
maxBatchSize = 50,
maxQueueSize = DEFAULT_MAX_QUEUE_SIZE,
failureThreshold = DEFAULT_FAILURE_THRESHOLD,
@@ -122,6 +132,8 @@ export class SQLBatcher {
) {
this.dbPool = dbPool;
this.maxTimeInQueue = maxTimeInQueue;
this.maxInFlightImmediate = maxInFlightImmediate;
this.#inFlight = inFlight;
this.maxBatchSize = maxBatchSize;
this.maxQueueSize = maxQueueSize;
this.failureThreshold = failureThreshold;
@@ -185,17 +197,34 @@ export class SQLBatcher {
if (this.queue.length >= this.maxBatchSize) {
this.flush(this.queue.splice(0, this.maxBatchSize));
} else if (this.queue.length === 1) {
this.timeouts.push(
setTimeout(() => {
this.flush(this.queue.splice(0, this.queue.length));
}, this.maxTimeInQueue),
);
} else if (this.#flushTimer === null) {
this.#armFlushTimer();
}
return promise;
}
// A lone flush holds a pooled connection for a full round trip, so only
// the first maxInFlightImmediate flushes skip the wait.
#armFlushTimer() {
const run = () => {
this.#flushTimer = null;
this.flush(this.queue.splice(0, this.queue.length));
};
this.#flushTimerIsImmediate =
this.#inFlight.count < this.maxInFlightImmediate;
this.#flushTimer = this.#flushTimerIsImmediate
? setImmediate(run)
: setTimeout(run, this.maxTimeInQueue);
}
#clearFlushTimer() {
if (this.#flushTimer === null) return;
if (this.#flushTimerIsImmediate) clearImmediate(this.#flushTimer);
else clearTimeout(this.#flushTimer);
this.#flushTimer = null;
}
// Bounded wait for a pooled connection. Without a bound, a stalled
// database turns every flush into an indefinite hang — nothing fails,
// so neither the breaker nor callers' own timeouts ever engage.
@@ -252,22 +281,23 @@ export class SQLBatcher {
}
async flush(batch) {
const timeout = this.timeouts.shift();
if (timeout && !timeout._destroyed) {
clearTimeout(timeout);
}
this.#clearFlushTimer();
if (batch.length === 0) return;
const query = `${batch.map((b) => b.sql.replace(/;+\s*$/, '')).join(';')}; SELECT 1`; // SELECT 1 forces mysql2 to return array
const values = batch.map((b) => b.values ?? []).flat();
this.#inFlight.count++;
try {
await this.#flushBatch(batch);
} finally {
this.#inFlight.count--;
}
}
async #flushBatch(batch) {
let connection;
try {
connection = await this.#acquireConnection();
} catch (error) {
this.#consecutiveFailures++;
this.#lastFailureAt = Date.now();
flushFailureCounter.add(1, this.#metricAttrs);
this.#recordFlushFailure();
console.warn(
'SQLBatcher could not acquire connection for flush:',
error,
@@ -278,6 +308,14 @@ export class SQLBatcher {
return;
}
if (batch.length === 1 && !this.readOnly) {
await this.#flushLoneWrite(batch[0], connection);
return;
}
const query = `${batch.map((b) => b.sql.replace(/;+\s*$/, '')).join(';')}; SELECT 1`; // SELECT 1 forces mysql2 to return array
const values = batch.map((b) => b.values ?? []).flat();
// Run the coalesced multi-statement inside an explicit transaction so
// a single bad statement (e.g. a duplicate-key INSERT) rolls back the
// whole batch atomically, leaving us free to re-run each item
@@ -333,7 +371,7 @@ export class SQLBatcher {
async () => {
while (cursor < batch.length) {
const i = cursor++;
settled[i] = await this.#runFallbackItem(batch[i]);
settled[i] = await this.#runItem(batch[i]);
}
},
);
@@ -367,7 +405,34 @@ export class SQLBatcher {
}
}
// Run one fallback item, retrying transient failures with backoff.
// A single statement is atomic under autocommit, so it skips BEGIN/COMMIT
// and their round trips. With no transaction to roll back, a failure is
// not re-run wholesale; only the per-item retry policy applies.
async #flushLoneWrite(item, connection) {
const r = await this.#runItem(item, connection);
if (r.ok) {
this.#consecutiveFailures = 0;
item.resolve(r.value);
return;
}
// An error the server answered with (duplicate key, bad SQL) means it
// is up; only failing to reach it counts toward the breaker.
if (isRetriableError(r.error)) {
this.#recordFlushFailure();
} else {
this.#consecutiveFailures = 0;
flushFailureCounter.add(1, this.#metricAttrs);
}
item.reject(r.error);
}
#recordFlushFailure() {
this.#consecutiveFailures++;
this.#lastFailureAt = Date.now();
flushFailureCounter.add(1, this.#metricAttrs);
}
// Run one item on its own, retrying transient failures with backoff.
// A read-only batcher may retry anything transient; a batcher that
// carries writes only retries failures where the statement provably
// did not apply — either it never reached the server, or the server
@@ -378,33 +443,37 @@ export class SQLBatcher {
// than surfacing: an item is a single statement, so a deadlock victim has
// been fully undone, and the caller sees an unhandled 500 for what the
// database is telling us to just run again.
async #runFallbackItem(b) {
async #runItem(b, connection) {
let attempt = 0;
while (true) {
let connection;
try {
connection = await this.#acquireConnection();
} catch (error) {
return { ok: false, error };
if (!connection) {
try {
connection = await this.#acquireConnection();
} catch (error) {
return { ok: false, error };
}
}
let error;
try {
return {
ok: true,
value: await connection.query(b.sql, b.values ?? []),
};
} catch (error) {
const canRetry = this.readOnly
? isRetriableError(error) || isRolledBackError(error)
: isNeverSentError(error) || isRolledBackError(error);
if (!canRetry || attempt >= ITEM_RETRY_ATTEMPTS) {
return { ok: false, error };
}
attempt++;
fallbackItemRetriesCounter.add(1, this.#metricAttrs);
await sleep(RETRY_BASE_BACKOFF_MS * attempt);
} catch (e) {
error = e;
} finally {
connection.release();
connection = undefined;
}
const canRetry = this.readOnly
? isRetriableError(error) || isRolledBackError(error)
: isNeverSentError(error) || isRolledBackError(error);
if (!canRetry || attempt >= ITEM_RETRY_ATTEMPTS) {
return { ok: false, error };
}
attempt++;
fallbackItemRetriesCounter.add(1, this.#metricAttrs);
await sleep(RETRY_BASE_BACKOFF_MS * attempt);
}
}
}
+410 -1
View File
@@ -69,6 +69,356 @@ const happyBatch = (sql: string) => {
];
};
// Each query on this connection waits until the test releases it, so its
// flush stays in flight.
const makeHeldConnection = () => {
const held: Array<() => void> = [];
const conn = makeConnection(
(sql) =>
new Promise((resolve) => {
held.push(() => resolve(happyBatch(sql)));
}),
);
const releaseAll = () => {
for (const release of held.splice(0)) release();
};
const sent = () => conn.query.mock.calls.map(([sql]) => sql);
return { conn, releaseAll, sent };
};
const fakeFlushTimers = () =>
vi.useFakeTimers({
toFake: [
'setTimeout',
'clearTimeout',
'setImmediate',
'clearImmediate',
],
});
describe('SQLBatcher flush timing', () => {
it('flushes a lone query on the next turn while under the in-flight cap', async () => {
const conn = makeConnection(happyBatch);
const { pool, getConnection } = makePool(conn);
const batcher = new SQLBatcher(pool, {
readOnly: true,
maxTimeInQueue: 60_000,
maxInFlightImmediate: 2,
});
const pending = batcher.query('SELECT a', []);
expect(getConnection).not.toHaveBeenCalled();
await new Promise((resolve) => setImmediate(resolve));
expect(getConnection).toHaveBeenCalledTimes(1);
expect((await pending)[0]).toEqual([{ n: 0 }]);
});
it('holds a lone query for maxTimeInQueue without an in-flight cap', async () => {
fakeFlushTimers();
try {
const conn = makeConnection(happyBatch);
const { pool, getConnection } = makePool(conn);
const batcher = new SQLBatcher(pool, {
readOnly: true,
maxTimeInQueue: 30,
});
const pending = batcher.query('SELECT a', []);
await vi.advanceTimersByTimeAsync(29);
expect(getConnection).not.toHaveBeenCalled();
await vi.advanceTimersByTimeAsync(1);
expect(getConnection).toHaveBeenCalledTimes(1);
expect((await pending)[0]).toEqual([{ n: 0 }]);
} finally {
vi.useRealTimers();
}
});
it('batches queries from the same turn and sends later ones separately', async () => {
const conn = makeConnection(happyBatch);
const { pool, getConnection } = makePool(conn);
const batcher = new SQLBatcher(pool, {
readOnly: true,
maxInFlightImmediate: 8,
});
const sameTurn = [
batcher.query('SELECT a', []),
batcher.query('SELECT b', []),
batcher.query('SELECT c', []),
];
await new Promise((resolve) => setImmediate(resolve));
const nextTurn = batcher.query('SELECT d', []);
const results = await Promise.all([...sameTurn, nextTurn]);
expect(results.map((r) => r[0])).toEqual([
[{ n: 0 }],
[{ n: 1 }],
[{ n: 2 }],
[{ n: 0 }],
]);
expect(getConnection).toHaveBeenCalledTimes(2);
expect(conn.query.mock.calls.map(([sql]) => sql)).toEqual([
'SELECT a;SELECT b;SELECT c; SELECT 1',
'SELECT d; SELECT 1',
]);
});
it('gives a query enqueued during an in-flight flush its own flush while under the cap', async () => {
const { conn, releaseAll, sent } = makeHeldConnection();
const { pool } = makePool(conn);
const batcher = new SQLBatcher(pool, {
readOnly: true,
maxInFlightImmediate: 2,
});
const turn = () => new Promise((resolve) => setImmediate(resolve));
const a = batcher.query('SELECT a', []);
await turn();
const b = batcher.query('SELECT b', []);
await turn();
expect(sent()).toEqual(['SELECT a; SELECT 1', 'SELECT b; SELECT 1']);
releaseAll();
expect((await a)[0]).toEqual([{ n: 0 }]);
expect((await b)[0]).toEqual([{ n: 0 }]);
});
it('holds a lone query for maxTimeInQueue at the in-flight cap and coalesces what follows', async () => {
fakeFlushTimers();
try {
const { conn, releaseAll, sent } = makeHeldConnection();
const { pool } = makePool(conn);
const batcher = new SQLBatcher(pool, {
readOnly: true,
maxTimeInQueue: 30,
maxInFlightImmediate: 1,
});
const a = batcher.query('SELECT a', []);
await vi.advanceTimersByTimeAsync(0);
expect(sent()).toEqual(['SELECT a; SELECT 1']);
const b = batcher.query('SELECT b', []);
await vi.advanceTimersByTimeAsync(29);
const c = batcher.query('SELECT c', []);
expect(sent()).toHaveLength(1);
await vi.advanceTimersByTimeAsync(1);
expect(sent()).toEqual([
'SELECT a; SELECT 1',
'SELECT b;SELECT c; SELECT 1',
]);
releaseAll();
expect((await a)[0]).toEqual([{ n: 0 }]);
expect((await b)[0]).toEqual([{ n: 0 }]);
expect((await c)[0]).toEqual([{ n: 1 }]);
// Settled flushes free the cap again.
const d = batcher.query('SELECT d', []);
await vi.advanceTimersByTimeAsync(0);
expect(sent()).toHaveLength(3);
releaseAll();
expect((await d)[0]).toEqual([{ n: 0 }]);
} finally {
vi.useRealTimers();
}
});
it('shares the in-flight cap between batchers given one counter', async () => {
fakeFlushTimers();
try {
const { conn, releaseAll, sent } = makeHeldConnection();
const { pool } = makePool(conn);
const inFlight = { count: 0 };
const opts = {
readOnly: true,
maxTimeInQueue: 30,
maxInFlightImmediate: 1,
inFlight,
};
const first = new SQLBatcher(pool, opts);
const second = new SQLBatcher(pool, opts);
const own = new SQLBatcher(pool, { ...opts, inFlight: undefined });
const a = first.query('SELECT a', []);
await vi.advanceTimersByTimeAsync(0);
const b = second.query('SELECT b', []);
const c = own.query('SELECT c', []);
await vi.advanceTimersByTimeAsync(0);
expect(sent()).toEqual([
'SELECT a; SELECT 1',
'SELECT c; SELECT 1',
]);
await vi.advanceTimersByTimeAsync(30);
expect(sent()).toHaveLength(3);
expect(sent()[2]).toBe('SELECT b; SELECT 1');
releaseAll();
await Promise.all([a, b, c]);
expect(inFlight.count).toBe(0);
} finally {
vi.useRealTimers();
}
});
it('counts a flush as in flight until its per-item fallback settles', async () => {
const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {});
fakeFlushTimers();
try {
const held: Array<() => void> = [];
const conn = makeConnection((sql) => {
if (isBatchQuery(sql)) throw makeError('ECONNRESET');
return new Promise((resolve) => {
held.push(() => resolve([[{ ok: sql }], undefined]));
});
});
const { pool } = makePool(conn);
const sent = () => conn.query.mock.calls.map(([sql]) => sql);
const batcher = new SQLBatcher(pool, {
readOnly: true,
maxTimeInQueue: 30,
maxInFlightImmediate: 1,
acquireTimeoutMs: 0,
});
const a = batcher.query('SELECT a', []);
await vi.advanceTimersByTimeAsync(0);
// The batch failed; the per-item retry of `a` is still running.
expect(sent()).toEqual(['SELECT a; SELECT 1', 'SELECT a']);
const b = batcher.query('SELECT b', []);
await vi.advanceTimersByTimeAsync(29);
expect(sent()).toHaveLength(2);
await vi.advanceTimersByTimeAsync(1);
expect(sent()).toEqual([
'SELECT a; SELECT 1',
'SELECT a',
'SELECT b; SELECT 1',
'SELECT b',
]);
for (const release of held.splice(0)) release();
expect((await a)[0]).toEqual([{ ok: 'SELECT a' }]);
expect((await b)[0]).toEqual([{ ok: 'SELECT b' }]);
} finally {
vi.useRealTimers();
warnSpy.mockRestore();
}
});
it('flushes a full batch at once past the cap and cancels the armed wait', async () => {
fakeFlushTimers();
try {
const { conn, releaseAll, sent } = makeHeldConnection();
const { pool, getConnection } = makePool(conn);
const batcher = new SQLBatcher(pool, {
readOnly: true,
maxBatchSize: 2,
maxTimeInQueue: 30,
maxInFlightImmediate: 1,
acquireTimeoutMs: 0,
});
const a = batcher.query('SELECT a', []);
await vi.advanceTimersByTimeAsync(0);
const b = batcher.query('SELECT b', []);
expect(vi.getTimerCount()).toBe(1);
const c = batcher.query('SELECT c', []);
expect(getConnection).toHaveBeenCalledTimes(2);
expect(vi.getTimerCount()).toBe(0);
// A later lone query waits its own full maxTimeInQueue.
await vi.advanceTimersByTimeAsync(10);
const d = batcher.query('SELECT d', []);
await vi.advanceTimersByTimeAsync(29);
expect(getConnection).toHaveBeenCalledTimes(2);
await vi.advanceTimersByTimeAsync(1);
expect(getConnection).toHaveBeenCalledTimes(3);
expect(sent()).toEqual([
'SELECT a; SELECT 1',
'SELECT b;SELECT c; SELECT 1',
'SELECT d; SELECT 1',
]);
releaseAll();
await Promise.all([a, b, c, d]);
} finally {
vi.useRealTimers();
}
});
it('cancels the pending next-turn flush when a batch fills up', async () => {
fakeFlushTimers();
try {
const conn = makeConnection(happyBatch);
const { pool, getConnection } = makePool(conn);
const batcher = new SQLBatcher(pool, {
maxBatchSize: 2,
readOnly: true,
maxInFlightImmediate: 8,
acquireTimeoutMs: 0,
});
const a = batcher.query('SELECT a', []);
expect(vi.getTimerCount()).toBe(1);
const b = batcher.query('SELECT b', []);
expect(vi.getTimerCount()).toBe(0);
const c = batcher.query('SELECT c', []);
expect(vi.getTimerCount()).toBe(1);
await vi.runAllTimersAsync();
await Promise.all([a, b, c]);
expect(vi.getTimerCount()).toBe(0);
expect(getConnection).toHaveBeenCalledTimes(2);
} finally {
vi.useRealTimers();
}
});
it('leaves no timer armed after a failed flush, a drop or a breaker rejection', async () => {
const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {});
fakeFlushTimers();
try {
const getConnection = vi.fn(async () => {
throw makeError('ECONNREFUSED');
});
const batcher = new SQLBatcher(
{ promise: () => ({ getConnection }) },
{
readOnly: true,
maxQueueSize: 1,
failureThreshold: 1,
cooldownMs: 60_000,
maxInFlightImmediate: 8,
},
);
const dropped = expect(
batcher.query('SELECT a', []),
).rejects.toMatchObject({ reason: 'queueOverflow' });
const failed = expect(
batcher.query('SELECT b', []),
).rejects.toMatchObject({ reason: 'connAcquire' });
expect(vi.getTimerCount()).toBe(1);
await vi.runAllTimersAsync();
await dropped;
await failed;
expect(vi.getTimerCount()).toBe(0);
await expect(batcher.query('SELECT c', [])).rejects.toMatchObject({
reason: 'breakerOpen',
});
expect(vi.getTimerCount()).toBe(0);
expect(batcher.queue).toHaveLength(0);
} finally {
vi.useRealTimers();
warnSpy.mockRestore();
}
});
});
describe('SQLBatcher', () => {
it('resolves each batched item with its own result', async () => {
const conn = makeConnection(happyBatch);
@@ -126,7 +476,7 @@ describe('SQLBatcher', () => {
});
it('drops the oldest item with reason queueOverflow at the high-water mark', async () => {
const conn = makeConnection(happyBatch);
const conn = makeConnection(() => [[{ n: 0 }], undefined]);
const { pool } = makePool(conn);
const batcher = new SQLBatcher(pool, {
maxTimeInQueue: 5,
@@ -277,6 +627,65 @@ describe('SQLBatcher', () => {
);
});
it('sends a lone write as-is, outside a transaction', async () => {
const conn = makeConnection((sql) => {
if (isBatchQuery(sql)) throw new Error('unexpected batch query');
return [{ affectedRows: 1 }, undefined];
});
const { pool } = makePool(conn);
const batcher = new SQLBatcher(pool, { maxBatchSize: 1 });
const result = await batcher.query('UPDATE t SET x = ?', [1]);
expect(result[0]).toEqual({ affectedRows: 1 });
expect(conn.query).toHaveBeenCalledTimes(1);
expect(conn.query).toHaveBeenCalledWith('UPDATE t SET x = ?', [1]);
expect(conn.beginTransaction).not.toHaveBeenCalled();
expect(conn.commit).not.toHaveBeenCalled();
expect(conn.release).toHaveBeenCalledTimes(1);
});
it('runs a lone write the server rejects once, without tripping the breaker', async () => {
const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {});
const conn = makeConnection((sql) => {
if (sql === 'INSERT dup') throw makeError('ER_DUP_ENTRY');
return [{ affectedRows: 1 }, undefined];
});
const { pool } = makePool(conn);
const batcher = new SQLBatcher(pool, {
maxBatchSize: 1,
failureThreshold: 1,
cooldownMs: 60_000,
});
await expect(batcher.query('INSERT dup', [])).rejects.toMatchObject({
code: 'ER_DUP_ENTRY',
});
expect(conn.query).toHaveBeenCalledTimes(1);
expect(conn.rollback).not.toHaveBeenCalled();
expect(warnSpy).not.toHaveBeenCalled();
// The server answered, so it is up: the next write goes through.
await expect(batcher.query('INSERT fine', [])).resolves.toBeTruthy();
warnSpy.mockRestore();
});
it('retries a lone write on a connection the server closed for inactivity', async () => {
let attempts = 0;
const conn = makeConnection(() => {
attempts++;
if (attempts === 1) {
throw makeError('ER_CLIENT_INTERACTION_TIMEOUT');
}
return [{ affectedRows: 1 }, undefined];
});
const { pool, getConnection } = makePool(conn);
const batcher = new SQLBatcher(pool, { maxBatchSize: 1 });
const result = await batcher.query('INSERT x', []);
expect(result[0]).toEqual({ affectedRows: 1 });
expect(attempts).toBe(2);
expect(getConnection).toHaveBeenCalledTimes(2);
});
it('never retries deterministic row-level errors and does not escalate the breaker', async () => {
let fallbackAttempts = 0;
const conn = makeConnection((sql) => {
@@ -27,7 +27,7 @@ import { DatabaseClientFactory } from './index.js';
import { SqliteDatabaseClient } from './SqliteDatabaseClient.js';
/** Highest schema version the migration table can reach. */
const CURRENT_SCHEMA_VERSION = 82;
const CURRENT_SCHEMA_VERSION = 91;
/**
* These suites migrate real files on disk. Idle they finish in well under a
@@ -930,6 +930,7 @@ describe('SqliteDatabaseClient — query interface', () => {
affectedRows: 1,
anyRowsAffected: true,
});
expect(inserted).not.toHaveProperty('warningStatus');
const missed = await client.write(
'UPDATE `widget` SET `name` = ? WHERE `id` = ?',
@@ -1038,8 +1039,7 @@ describe('SqliteDatabaseClient — query interface', () => {
});
// Single-node engines have no replica to race, so `tryHardRead` must not
// issue the query twice — the base-class default fires `pread()` and
// `read()` in parallel, which on sqlite is the same connection.
// issue the query twice on the one connection.
describe('SqliteDatabaseClient — tryHardRead', () => {
let client: SqliteDatabaseClient;
@@ -1078,20 +1078,50 @@ describe('SqliteDatabaseClient — tryHardRead', () => {
expect(readSpy).toHaveBeenCalledTimes(1);
expect(preadSpy).not.toHaveBeenCalled();
});
});
it('throws from requireRead when nothing matches', async () => {
await expect(
client.requireRead(
'SELECT `name` FROM `widget` WHERE `id` = ?',
[42],
),
).rejects.toThrow('required read returned no rows');
describe('SqliteDatabaseClient — prepared statements', () => {
let client: SqliteDatabaseClient;
let prepare: ReturnType<typeof vi.spyOn>;
beforeEach(async () => {
client = await bootClient();
await client.write(
'CREATE TABLE `widget` (`id` INTEGER PRIMARY KEY, `name` TEXT)',
);
// better-sqlite3 is the driver boundary; count compilations there.
const db = (client as unknown as { db: Database.Database }).db;
prepare = vi.spyOn(db, 'prepare');
});
it('returns the rows from requireRead when something matches', async () => {
await client.write('INSERT INTO `widget` (`name`) VALUES (?)', ['ok']);
await expect(
client.requireRead('SELECT `name` FROM `widget`'),
).resolves.toEqual([{ name: 'ok' }]);
afterEach(() => {
client.onServerShutdown();
});
it('compiles a repeated query once', async () => {
const insert = 'INSERT INTO `widget` (`name`) VALUES (?)';
await client.write(insert, ['a']);
await client.write(insert, ['b']);
await client.batchWrite([{ statement: insert, values: ['c'] }]);
const select = 'SELECT `name` FROM `widget` ORDER BY `id`';
await client.read(select);
await expect(client.read(select)).resolves.toEqual([
{ name: 'a' },
{ name: 'b' },
{ name: 'c' },
]);
expect(prepare).toHaveBeenCalledTimes(2);
});
it('keeps serving a cached query after the schema changes', async () => {
const select = 'SELECT * FROM `widget`';
await client.write('INSERT INTO `widget` (`name`) VALUES (?)', ['a']);
await client.read(select);
await client.write('ALTER TABLE `widget` ADD COLUMN `size` INTEGER');
await expect(client.read(select)).resolves.toEqual([
{ id: 1, name: 'a', size: null },
]);
});
});
@@ -21,10 +21,15 @@ import { existsSync, mkdirSync, readFileSync } from 'fs';
import { basename, dirname, extname, join, resolve } from 'path';
import { createContext, runInContext } from 'vm';
import type { IConfig } from '../../types';
import { BoundedTtlMap } from '../../util/boundedTtlMap.js';
import { Span } from '../../util/span.js';
import { AbstractDatabaseClient, type WriteResult } from './DatabaseClient';
const MIGRATIONS_DIR = resolve(__dirname, './migrations/sqlite');
const STATEMENT_CACHE_SIZE = 1000;
type SqliteDatabase = InstanceType<typeof import('better-sqlite3')>;
type SqliteStatement = import('better-sqlite3').Statement<unknown[]>;
/**
* Ordered list of [threshold_version, files[]] pairs. A database whose
@@ -116,13 +121,26 @@ const AVAILABLE_MIGRATIONS: [number, string[]][] = [
[79, ['0084_share-anyone-with-link.sql']],
[80, ['0085_event-subscriptions-include-value.sql']],
[81, ['0086_subdomains-app-owner-cascade.sql']],
[82, ['0087_feedback-attachments.sql']],
[83, ['0088_apps-index-url.sql']],
[84, ['0089_team-require-2fa.sql']],
[85, ['0090_user-app-permission-index.sql']],
[86, ['0091_icloud-dot-significance.sql']],
[87, ['0092_share-entry-owner.sql']],
[88, ['0093_fsentries-parent-uid-index.sql']],
[89, ['0094_deleted-apps.sql']],
[90, ['0095_kv-share-handle-levels.sql']],
];
export class SqliteDatabaseClient extends AbstractDatabaseClient {
override readonly engineName = 'sqlite';
// better-sqlite3 instance — set during onServerStart
private db!: InstanceType<typeof import('better-sqlite3')>;
private db!: SqliteDatabase;
/** Prepared statements by query text; SQLite re-prepares on schema change. */
private readonly statements = new BoundedTtlMap<string, SqliteStatement>({
maxEntries: STATEMENT_CACHE_SIZE,
});
constructor(config: IConfig) {
super(config);
@@ -144,12 +162,14 @@ export class SqliteDatabaseClient extends AbstractDatabaseClient {
mkdirSync(dirname(dbPath), { recursive: true });
}
this.statements.clear();
this.db = new Database(dbPath);
await this.runMigrations(isNew);
}
override onServerShutdown(): void {
this.statements.clear();
if (this.db) {
this.db.close();
}
@@ -164,12 +184,9 @@ export class SqliteDatabaseClient extends AbstractDatabaseClient {
query: string,
params: unknown[] = [],
): Promise<Record<string, unknown>[]> {
query = this.transformQuery(query);
params = this.transformParams(params);
return this.db.prepare(query).all(...params) as Record<
string,
unknown
>[];
return this.prepare(query).all(
...this.transformParams(params),
) as Record<string, unknown>[];
}
override async pread(
@@ -180,30 +197,20 @@ export class SqliteDatabaseClient extends AbstractDatabaseClient {
return this.read(query, params);
}
override async tryHardRead(
query: string,
params: unknown[] = [],
): Promise<Record<string, unknown>[]> {
// No replica to race, so the base class's parallel
// primary-plus-replica read would run the same statement twice on
// the one connection.
return this.read(query, params);
}
@Span('db.write', (query: string) => ({ 'db.statement': query }))
override async write(
query: string,
params: unknown[] = [],
): Promise<WriteResult> {
query = this.transformQuery(query);
params = this.transformParams(params);
const info = this.db.prepare(query).run(...params);
const info = this.prepare(query).run(...this.transformParams(params));
return {
insertId: info.lastInsertRowid,
affectedRows: info.changes,
anyRowsAffected: info.changes > 0,
// No warningStatus: SQLite raises none, but a new row can reuse a
// deleted row's rowid that other tables still name, so a clean
// insert doesn't prove the row is all the caller sent.
};
}
@@ -214,10 +221,8 @@ export class SqliteDatabaseClient extends AbstractDatabaseClient {
entries: { statement: string; values: unknown[] }[],
): Promise<void> {
this.db.transaction(() => {
for (let { statement, values } of entries) {
statement = this.transformQuery(statement);
values = this.transformParams(values);
this.db.prepare(statement).run(...values);
for (const { statement, values } of entries) {
this.prepare(statement).run(...this.transformParams(values));
}
})();
}
@@ -226,6 +231,15 @@ export class SqliteDatabaseClient extends AbstractDatabaseClient {
// SQLite-specific transforms
// ------------------------------------------------------------------
private prepare(query: string): SqliteStatement {
let statement = this.statements.get(query);
if (!statement) {
statement = this.db.prepare(this.transformQuery(query));
this.statements.set(query, statement);
}
return statement;
}
private transformQuery(query: string): string {
return query.replace(/now\(\)/gi, "datetime('now')");
}
@@ -0,0 +1,39 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Mirrors SQLite migration 0087: Contact Us attachment metadata, a JSON array
-- of `{name, type, size}` (NULL when none). Guarded by a throwaway procedure,
-- as in mysql_mig_21.
DROP PROCEDURE IF EXISTS _puter_add_feedback_attachments;
DELIMITER //
CREATE PROCEDURE _puter_add_feedback_attachments()
BEGIN
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'feedback'
AND COLUMN_NAME = 'attachments'
) THEN
ALTER TABLE `feedback` ADD COLUMN `attachments` text DEFAULT NULL;
END IF;
END//
DELIMITER ;
CALL _puter_add_feedback_attachments();
DROP PROCEDURE IF EXISTS _puter_add_feedback_attachments;
@@ -0,0 +1,44 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Mirrors SQLite migration 0088: index `apps.index_url` for origin lookups.
-- TEXT needs a prefix; 191 characters keeps the key under 767 bytes in
-- utf8mb4 whatever the table's row format, and URLs diverge well before it.
--
-- Idempotent: the guarded procedure, as mig_36. The 1061 handler covers two
-- nodes booting at once, where the loser finds the index already built.
DROP PROCEDURE IF EXISTS _puter_add_apps_index_url_index;
DELIMITER //
CREATE PROCEDURE _puter_add_apps_index_url_index()
BEGIN
DECLARE CONTINUE HANDLER FOR 1061 BEGIN END;
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'apps'
AND INDEX_NAME = 'idx_apps_index_url'
) THEN
ALTER TABLE `apps`
ADD INDEX `idx_apps_index_url` (`index_url`(191)),
ALGORITHM=INPLACE, LOCK=NONE;
END IF;
END//
DELIMITER ;
CALL _puter_add_apps_index_url_index();
DROP PROCEDURE IF EXISTS _puter_add_apps_index_url_index;
@@ -0,0 +1,21 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Mirrors SQLite migration 0089: a team may require 2FA of the accounts it
-- provisioned. Added through `_puter_add_col` as mysql_mig_32 does, so a
-- replay finds the column and does nothing.
CALL _puter_add_col('group', 'require_2fa', '`require_2fa` tinyint(1) NOT NULL DEFAULT 0');
@@ -0,0 +1,39 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Mirrors SQLite migration 0090. Guarded like mysql_mig_22, because there is
-- no per-file applied-state tracking and a replay must do nothing.
DROP PROCEDURE IF EXISTS _puter_add_user_app_permission_index;
DELIMITER //
CREATE PROCEDURE _puter_add_user_app_permission_index()
BEGIN
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'user_to_app_permissions'
AND INDEX_NAME = 'idx_user_to_app_permissions_permission'
) THEN
ALTER TABLE `user_to_app_permissions`
ADD INDEX `idx_user_to_app_permissions_permission` (`permission`);
END IF;
END//
DELIMITER ;
CALL _puter_add_user_app_permission_index();
DROP PROCEDURE IF EXISTS _puter_add_user_app_permission_index;
@@ -0,0 +1,49 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Recompute `clean_email` for Apple's domains: dots are significant there, so
-- rows written while they were stripped still collapse two mailboxes into one.
-- Only rows still holding exactly what the old rule produced, so a NULL or a
-- legacy un-lowercased value is left alone -- rewriting those would fold `+tag`
-- or case into a key another row already owns. Restores dots and nothing else.
UPDATE `user`
SET `clean_email` = CONCAT(SUBSTRING_INDEX(SUBSTRING_INDEX(LOWER(`email`), '@', 1), '+', 1), '@', SUBSTRING_INDEX(LOWER(`email`), '@', -1))
WHERE `email` IS NOT NULL
AND LOCATE('@', `email`) > 1
AND (
LOWER(`email`) LIKE '%@icloud.com'
OR LOWER(`email`) LIKE '%@me.com'
OR LOWER(`email`) LIKE '%@mac.com'
)
AND `clean_email` = CONCAT(REPLACE(SUBSTRING_INDEX(SUBSTRING_INDEX(LOWER(`email`), '@', 1), '+', 1), '.', ''), '@', SUBSTRING_INDEX(LOWER(`email`), '@', -1))
AND `clean_email` <> CONCAT(SUBSTRING_INDEX(SUBSTRING_INDEX(LOWER(`email`), '@', 1), '+', 1), '@', SUBSTRING_INDEX(LOWER(`email`), '@', -1));
-- The same collapse decided who a pending invite belongs to, so the holder
-- of the undotted address claims invites addressed to the dotted one. The
-- typed address rides along as `invitedAddress` whenever it differed.
UPDATE `share`
SET `recipient_email` = CONCAT(SUBSTRING_INDEX(SUBSTRING_INDEX(LOWER(JSON_UNQUOTE(JSON_EXTRACT(`data`, '$.invitedAddress'))), '@', 1), '+', 1), '@', SUBSTRING_INDEX(LOWER(JSON_UNQUOTE(JSON_EXTRACT(`data`, '$.invitedAddress'))), '@', -1))
WHERE JSON_UNQUOTE(JSON_EXTRACT(`data`, '$.invitedAddress')) IS NOT NULL
AND LOCATE('@', JSON_UNQUOTE(JSON_EXTRACT(`data`, '$.invitedAddress'))) > 1
AND (
LOWER(`recipient_email`) LIKE '%@icloud.com'
OR LOWER(`recipient_email`) LIKE '%@me.com'
OR LOWER(`recipient_email`) LIKE '%@mac.com'
)
AND `recipient_email` = CONCAT(REPLACE(SUBSTRING_INDEX(SUBSTRING_INDEX(LOWER(JSON_UNQUOTE(JSON_EXTRACT(`data`, '$.invitedAddress'))), '@', 1), '+', 1), '.', ''), '@', SUBSTRING_INDEX(LOWER(JSON_UNQUOTE(JSON_EXTRACT(`data`, '$.invitedAddress'))), '@', -1))
AND `recipient_email` <> CONCAT(SUBSTRING_INDEX(SUBSTRING_INDEX(LOWER(JSON_UNQUOTE(JSON_EXTRACT(`data`, '$.invitedAddress'))), '@', 1), '+', 1), '@', SUBSTRING_INDEX(LOWER(JSON_UNQUOTE(JSON_EXTRACT(`data`, '$.invitedAddress'))), '@', -1));
@@ -0,0 +1,52 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- The owner of the shared entry, recorded on the share row. Mirrors SQLite
-- migration 0092. Guarded like mysql_mig_45, because there is no per-file
-- applied-state tracking and a replay must do nothing.
DROP PROCEDURE IF EXISTS _puter_add_share_entry_owner;
DELIMITER //
CREATE PROCEDURE _puter_add_share_entry_owner()
BEGIN
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'share'
AND COLUMN_NAME = 'entry_owner_user_id'
) THEN
ALTER TABLE `share` ADD COLUMN `entry_owner_user_id` BIGINT DEFAULT NULL;
END IF;
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'share'
AND INDEX_NAME = 'idx_share_entry_owner'
) THEN
ALTER TABLE `share` ADD INDEX `idx_share_entry_owner` (`entry_owner_user_id`, `id`);
END IF;
END//
DELIMITER ;
CALL _puter_add_share_entry_owner();
DROP PROCEDURE IF EXISTS _puter_add_share_entry_owner;
UPDATE `share` JOIN `fsentries` ON `fsentries`.`id` = `share`.`fsentry_id`
SET `share`.`entry_owner_user_id` = `fsentries`.`user_id`
WHERE `share`.`fsentry_id` IS NOT NULL
AND `share`.`entry_owner_user_id` IS NULL;
@@ -0,0 +1,26 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Uid and owner of each deleted app. Mirrors SQLite migration 0094.
--
-- Idempotent: `CREATE TABLE IF NOT EXISTS` lets the directory replay safely.
CREATE TABLE IF NOT EXISTS `deleted_apps` (
`uid` VARCHAR(40) NOT NULL,
`owner_user_id` INT UNSIGNED DEFAULT NULL,
PRIMARY KEY (`uid`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
@@ -0,0 +1,50 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Share levels on `kv_share_handles`. See
-- sqlite/0095_kv-share-handle-levels.sql for the rationale. No per-file
-- applied-state tracking, so every step is guarded and a replay does nothing.
CALL _puter_add_col('kv_share_handles', 'revoked_by', '`revoked_by` varchar(8) DEFAULT NULL');
CALL _puter_add_col('kv_share_handles', 'label', '`label` varchar(512) DEFAULT NULL');
DROP PROCEDURE IF EXISTS _puter_add_kv_share_handle_indexes;
DELIMITER //
CREATE PROCEDURE _puter_add_kv_share_handle_indexes()
BEGIN
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'kv_share_handles'
AND INDEX_NAME = 'idx_kv_share_handles_grantee'
) THEN
ALTER TABLE `kv_share_handles` ADD INDEX `idx_kv_share_handles_grantee` (`grantee_user_id`, `id`);
END IF;
IF NOT EXISTS (
SELECT 1 FROM INFORMATION_SCHEMA.STATISTICS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'kv_share_handles'
AND INDEX_NAME = 'idx_kv_share_handles_app'
) THEN
ALTER TABLE `kv_share_handles` ADD INDEX `idx_kv_share_handles_app` (`app_uid`);
END IF;
END//
DELIMITER ;
CALL _puter_add_kv_share_handle_indexes();
DROP PROCEDURE IF EXISTS _puter_add_kv_share_handle_indexes;
@@ -0,0 +1,21 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Mirrors SQLite migration 0087: Contact Us attachment metadata, a JSON array
-- of `{name, type, size}` (NULL when none).
ALTER TABLE feedback ADD COLUMN IF NOT EXISTS attachments text DEFAULT NULL;
@@ -0,0 +1,24 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Mirrors SQLite migration 0088: index `apps.index_url` for origin lookups.
-- Hash rather than btree: `index_url` allows 3000 characters, past btree's
-- ~2.7 KB row limit, and every lookup is equality (`=` / `IN`).
--
-- Idempotent via IF NOT EXISTS.
CREATE INDEX IF NOT EXISTS idx_apps_index_url ON apps USING hash (index_url);
@@ -0,0 +1,21 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Mirrors SQLite migration 0089: a team may require 2FA of the accounts it
-- provisioned. Idempotent -- there is no per-file applied-state tracking, so
-- a replay must find the column already present and do nothing.
ALTER TABLE "group" ADD COLUMN IF NOT EXISTS require_2fa smallint NOT NULL DEFAULT 0;
@@ -0,0 +1,24 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Mirrors SQLite migration 0090, as postgres_mig_11 did for user_to_user.
-- text_pattern_ops is what makes the left-anchored LIKE a range scan under a
-- non-C collation -- the plain index dev_to_app already has does not.
CREATE INDEX IF NOT EXISTS idx_user_to_app_permissions_permission
ON user_to_app_permissions (permission text_pattern_ops);
CREATE INDEX IF NOT EXISTS idx_dev_to_app_permissions_permission_pattern
ON dev_to_app_permissions (permission text_pattern_ops);
@@ -0,0 +1,49 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Recompute `clean_email` for Apple's domains: dots are significant there, so
-- rows written while they were stripped still collapse two mailboxes into one.
-- Only rows still holding exactly what the old rule produced, so a NULL or a
-- legacy un-lowercased value is left alone -- rewriting those would fold `+tag`
-- or case into a key another row already owns. Restores dots and nothing else.
UPDATE "user"
SET clean_email = (split_part(split_part(lower(email), '@', 1), '+', 1) || '@' || split_part(lower(email), '@', 2))
WHERE email IS NOT NULL
AND position('@' in email) > 1
AND (
lower(email) LIKE '%@icloud.com'
OR lower(email) LIKE '%@me.com'
OR lower(email) LIKE '%@mac.com'
)
AND clean_email = (replace(split_part(split_part(lower(email), '@', 1), '+', 1), '.', '') || '@' || split_part(lower(email), '@', 2))
AND clean_email IS DISTINCT FROM (split_part(split_part(lower(email), '@', 1), '+', 1) || '@' || split_part(lower(email), '@', 2));
-- The same collapse decided who a pending invite belongs to, so the holder
-- of the undotted address claims invites addressed to the dotted one. The
-- typed address rides along as `invitedAddress` whenever it differed.
UPDATE "share"
SET recipient_email = (split_part(split_part(lower((data->>'invitedAddress')), '@', 1), '+', 1) || '@' || split_part(lower((data->>'invitedAddress')), '@', 2))
WHERE (data->>'invitedAddress') IS NOT NULL
AND position('@' in (data->>'invitedAddress')) > 1
AND (
lower(recipient_email) LIKE '%@icloud.com'
OR lower(recipient_email) LIKE '%@me.com'
OR lower(recipient_email) LIKE '%@mac.com'
)
AND recipient_email = (replace(split_part(split_part(lower((data->>'invitedAddress')), '@', 1), '+', 1), '.', '') || '@' || split_part(lower((data->>'invitedAddress')), '@', 2))
AND recipient_email IS DISTINCT FROM (split_part(split_part(lower((data->>'invitedAddress')), '@', 1), '+', 1) || '@' || split_part(lower((data->>'invitedAddress')), '@', 2));
@@ -0,0 +1,32 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- The owner of the shared entry, recorded on the share row. Mirrors SQLite
-- migration 0092. Guarded like mysql_mig_45, because there is no per-file
-- applied-state tracking and a replay must do nothing.
ALTER TABLE "share" ADD COLUMN IF NOT EXISTS entry_owner_user_id BIGINT DEFAULT NULL;
CREATE INDEX IF NOT EXISTS idx_share_entry_owner
ON "share" (entry_owner_user_id, id);
UPDATE "share"
SET entry_owner_user_id = f.user_id
FROM fsentries f
WHERE f.id = "share".fsentry_id
AND "share".fsentry_id IS NOT NULL
AND "share".entry_owner_user_id IS NULL;
@@ -0,0 +1,26 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Uid and owner of each deleted app. Mirrors SQLite migration 0094 / MySQL
-- mysql_mig_48.
--
-- Idempotent via IF NOT EXISTS.
CREATE TABLE IF NOT EXISTS deleted_apps (
uid VARCHAR(40) PRIMARY KEY,
owner_user_id INTEGER DEFAULT NULL
);
@@ -0,0 +1,29 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Share levels on kv_share_handles. See
-- sqlite/0095_kv-share-handle-levels.sql for the rationale. Idempotent via
-- IF NOT EXISTS; there is no per-file applied-state tracking.
ALTER TABLE kv_share_handles ADD COLUMN IF NOT EXISTS revoked_by VARCHAR(8) DEFAULT NULL;
ALTER TABLE kv_share_handles ADD COLUMN IF NOT EXISTS label VARCHAR(512) DEFAULT NULL;
CREATE INDEX IF NOT EXISTS idx_kv_share_handles_grantee
ON kv_share_handles (grantee_user_id, id);
CREATE INDEX IF NOT EXISTS idx_kv_share_handles_app
ON kv_share_handles (app_uid);
@@ -0,0 +1,21 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Contact Us attachment metadata: a JSON array of `{name, type, size}`, NULL
-- when none. The files themselves go out with the support email.
ALTER TABLE `feedback` ADD COLUMN "attachments" TEXT DEFAULT NULL;
@@ -0,0 +1,22 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Resolving an origin to its app looks `apps` up by `index_url` on every
-- origin-based app token request (`AuthService#findCanonicalAppUidForOrigin`),
-- plus `AppStore.existsByIndexUrl`. With no index each lookup scans the table.
CREATE INDEX IF NOT EXISTS `idx_apps_index_url` ON `apps` (`index_url`);
@@ -0,0 +1,21 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Whether every account this team provisioned must hold 2FA before it can
-- use Puter. Off until the owner turns it on, and the owner must hold 2FA
-- themselves first, or enabling it would lock out the account that did so.
ALTER TABLE `group` ADD COLUMN `require_2fa` INTEGER NOT NULL DEFAULT 0;
@@ -0,0 +1,26 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Withdrawing an app's cross-app data grants looks them up by permission text,
-- and no existing index on either table leads with `permission`. Same fix as
-- `idx_user_to_user_permissions_permission` in 0067.
CREATE INDEX IF NOT EXISTS `idx_user_to_app_permissions_permission`
ON `user_to_app_permissions` (`permission`);
-- The sweep reads both tables; MySQL and Postgres already index this one.
CREATE INDEX IF NOT EXISTS `idx_dev_to_app_permissions_permission`
ON `dev_to_app_permissions` (`permission`);
@@ -0,0 +1,73 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Recompute `clean_email` for Apple's domains: dots are significant there, so
-- rows written while they were stripped still collapse two mailboxes into one.
-- Only rows still holding exactly what the old rule produced, so a NULL or a
-- legacy un-lowercased value is left alone -- rewriting those would fold `+tag`
-- or case into a key another row already owns. Restores dots and nothing else.
UPDATE `user`
SET `clean_email` = (CASE WHEN instr(substr(lower(`email`), 1, instr(`email`, '@') - 1), '+') > 0
THEN substr(substr(lower(`email`), 1, instr(`email`, '@') - 1), 1,
instr(substr(lower(`email`), 1, instr(`email`, '@') - 1), '+') - 1)
ELSE substr(lower(`email`), 1, instr(`email`, '@') - 1)
END || substr(lower(`email`), instr(`email`, '@')))
WHERE `email` IS NOT NULL
AND instr(`email`, '@') > 1
AND (
lower(`email`) LIKE '%@icloud.com'
OR lower(`email`) LIKE '%@me.com'
OR lower(`email`) LIKE '%@mac.com'
)
AND `clean_email` = (replace(CASE WHEN instr(substr(lower(`email`), 1, instr(`email`, '@') - 1), '+') > 0
THEN substr(substr(lower(`email`), 1, instr(`email`, '@') - 1), 1,
instr(substr(lower(`email`), 1, instr(`email`, '@') - 1), '+') - 1)
ELSE substr(lower(`email`), 1, instr(`email`, '@') - 1)
END, '.', '') || substr(lower(`email`), instr(`email`, '@')))
AND `clean_email` <> (CASE WHEN instr(substr(lower(`email`), 1, instr(`email`, '@') - 1), '+') > 0
THEN substr(substr(lower(`email`), 1, instr(`email`, '@') - 1), 1,
instr(substr(lower(`email`), 1, instr(`email`, '@') - 1), '+') - 1)
ELSE substr(lower(`email`), 1, instr(`email`, '@') - 1)
END || substr(lower(`email`), instr(`email`, '@')));
-- The same collapse decided who a pending invite belongs to, so the holder
-- of the undotted address claims invites addressed to the dotted one. The
-- typed address rides along as `invitedAddress` whenever it differed.
UPDATE `share`
SET `recipient_email` = (CASE WHEN instr(substr(lower(json_extract(`data`, '$.invitedAddress')), 1, instr(json_extract(`data`, '$.invitedAddress'), '@') - 1), '+') > 0
THEN substr(substr(lower(json_extract(`data`, '$.invitedAddress')), 1, instr(json_extract(`data`, '$.invitedAddress'), '@') - 1), 1,
instr(substr(lower(json_extract(`data`, '$.invitedAddress')), 1, instr(json_extract(`data`, '$.invitedAddress'), '@') - 1), '+') - 1)
ELSE substr(lower(json_extract(`data`, '$.invitedAddress')), 1, instr(json_extract(`data`, '$.invitedAddress'), '@') - 1)
END || substr(lower(json_extract(`data`, '$.invitedAddress')), instr(json_extract(`data`, '$.invitedAddress'), '@')))
WHERE json_extract(`data`, '$.invitedAddress') IS NOT NULL
AND instr(json_extract(`data`, '$.invitedAddress'), '@') > 1
AND (
lower(`recipient_email`) LIKE '%@icloud.com'
OR lower(`recipient_email`) LIKE '%@me.com'
OR lower(`recipient_email`) LIKE '%@mac.com'
)
AND `recipient_email` = (replace(CASE WHEN instr(substr(lower(json_extract(`data`, '$.invitedAddress')), 1, instr(json_extract(`data`, '$.invitedAddress'), '@') - 1), '+') > 0
THEN substr(substr(lower(json_extract(`data`, '$.invitedAddress')), 1, instr(json_extract(`data`, '$.invitedAddress'), '@') - 1), 1,
instr(substr(lower(json_extract(`data`, '$.invitedAddress')), 1, instr(json_extract(`data`, '$.invitedAddress'), '@') - 1), '+') - 1)
ELSE substr(lower(json_extract(`data`, '$.invitedAddress')), 1, instr(json_extract(`data`, '$.invitedAddress'), '@') - 1)
END, '.', '') || substr(lower(json_extract(`data`, '$.invitedAddress')), instr(json_extract(`data`, '$.invitedAddress'), '@')))
AND `recipient_email` <> (CASE WHEN instr(substr(lower(json_extract(`data`, '$.invitedAddress')), 1, instr(json_extract(`data`, '$.invitedAddress'), '@') - 1), '+') > 0
THEN substr(substr(lower(json_extract(`data`, '$.invitedAddress')), 1, instr(json_extract(`data`, '$.invitedAddress'), '@') - 1), 1,
instr(substr(lower(json_extract(`data`, '$.invitedAddress')), 1, instr(json_extract(`data`, '$.invitedAddress'), '@') - 1), '+') - 1)
ELSE substr(lower(json_extract(`data`, '$.invitedAddress')), 1, instr(json_extract(`data`, '$.invitedAddress'), '@') - 1)
END || substr(lower(json_extract(`data`, '$.invitedAddress')), instr(json_extract(`data`, '$.invitedAddress'), '@')));
@@ -0,0 +1,40 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- The owner of the shared entry, recorded on the share row.
--
-- The outbound listings answer "shares on things I own that someone else
-- issued" by joining `share` to `fsentries` and filtering on the owner there.
-- Neither side of that join bounds the work: driven from `fsentries` it walks
-- every file the user owns, driven from `share` it walks every share until it
-- has a page. `#outboundAppsSql` has no LIMIT at all.
--
-- Written with the row, and re-pointed when a move re-owns the subtree it
-- names; `FSService.move` does that whenever the destination is someone
-- else's tree.
ALTER TABLE `share` ADD COLUMN `entry_owner_user_id` INTEGER DEFAULT NULL;
CREATE INDEX IF NOT EXISTS `idx_share_entry_owner`
ON `share` (`entry_owner_user_id`, `id`);
UPDATE `share`
SET `entry_owner_user_id` = (
SELECT `user_id` FROM `fsentries` WHERE `fsentries`.`id` = `share`.`fsentry_id`
)
WHERE `fsentry_id` IS NOT NULL
AND `entry_owner_user_id` IS NULL;
@@ -0,0 +1,22 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Directory listings and subtree walks (delete, copy) find children by
-- `parent_uid`, the one link every row carries; with no index each lookup
-- scans the table. MySQL and Postgres already index it.
CREATE INDEX IF NOT EXISTS `idx_fsentries_parent_uid`
ON `fsentries` (`parent_uid`);
@@ -0,0 +1,25 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- Uid and owner of each deleted app. An origin-derived uid comes back when the
-- origin is used again; on a hosted subdomain now registered to someone else,
-- AuthService moves on to a fresh uid so the deleted app's data stays put.
CREATE TABLE IF NOT EXISTS `deleted_apps` (
"uid" TEXT PRIMARY KEY NOT NULL,
"owner_user_id" INTEGER DEFAULT NULL
);
@@ -0,0 +1,35 @@
-- Copyright (C) 2024-present Puter Technologies Inc.
--
-- This file is part of Puter.
--
-- Puter is free software: you can redistribute it and/or modify
-- it under the terms of the GNU Affero General Public License as published
-- by the Free Software Foundation, either version 3 of the License, or
-- (at your option) any later version.
--
-- This program is distributed in the hope that it will be useful,
-- but WITHOUT ANY WARRANTY; without even the implied warranty of
-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-- GNU Affero General Public License for more details.
--
-- You should have received a copy of the GNU Affero General Public License
-- along with this program. If not, see <https://www.gnu.org/licenses/>.
-- What share levels need from `kv_share_handles`. The level itself is not a
-- column: it is the root of the stored `permission`.
--
-- - `revoked_by` : `owner`, `holder` or `system` (the namespace app was
-- deleted). Null while live, and on rows retired before
-- this existed.
-- - `label` : optional owner-set text the holder is shown.
-- - the grantee index serves the holder's listing and their unshare, and
-- the app index retiring a deleted app's handles.
ALTER TABLE `kv_share_handles` ADD COLUMN `revoked_by` TEXT DEFAULT NULL;
ALTER TABLE `kv_share_handles` ADD COLUMN `label` TEXT DEFAULT NULL;
CREATE INDEX IF NOT EXISTS `idx_kv_share_handles_grantee`
ON `kv_share_handles` (`grantee_user_id`, `id`);
CREATE INDEX IF NOT EXISTS `idx_kv_share_handles_app`
ON `kv_share_handles` (`app_uid`);
@@ -32,6 +32,7 @@ const RETRIABLE_ERROR_CODES = new Set([
'ENETUNREACH',
'EAI_AGAIN',
POOL_ACQUIRE_TIMEOUT,
'ER_CLIENT_INTERACTION_TIMEOUT',
]);
const RETRIABLE_ERROR_MESSAGES = [
@@ -52,6 +53,9 @@ const NEVER_SENT_ERROR_CODES = new Set([
'ENETUNREACH',
'EAI_AGAIN',
POOL_ACQUIRE_TIMEOUT,
// The server closed the pooled connection for inactivity before the
// statement arrived, so nothing ran.
'ER_CLIENT_INTERACTION_TIMEOUT',
]);
/**
+542 -1
View File
@@ -20,7 +20,17 @@
import { createServer, type Server } from 'node:http';
import type { AddressInfo } from 'node:net';
import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest';
import {
afterAll,
afterEach,
beforeAll,
beforeEach,
describe,
expect,
it,
type MockInstance,
vi,
} from 'vitest';
import type { IConfig } from '../../types';
import { DDBClient } from './DDBClient';
@@ -142,6 +152,83 @@ describe('DDBClient — item operations', () => {
expect(result.Item).toBeUndefined();
});
it('deletes only while its condition holds', async () => {
await client.put(TABLE, { pk: 'u6', sk: 'guarded', v: 1 });
await expect(
client.del(
TABLE,
{ pk: 'u6', sk: 'guarded' },
{
condition: {
expression: '#v = :wrong',
names: { '#v': 'v' },
values: { ':wrong': 2 },
},
},
),
).rejects.toMatchObject({ name: 'ConditionalCheckFailedException' });
expect(
(await client.get(TABLE, { pk: 'u6', sk: 'guarded' })).Item,
).toBeDefined();
await client.del(
TABLE,
{ pk: 'u6', sk: 'guarded' },
{
condition: {
expression: '#v = :right',
names: { '#v': 'v' },
values: { ':right': 1 },
},
},
);
expect(
(await client.get(TABLE, { pk: 'u6', sk: 'guarded' })).Item,
).toBeUndefined();
});
it('puts only while its condition holds', async () => {
const absent = {
expression: 'attribute_not_exists(#pk)',
names: { '#pk': 'pk' },
};
await client.put(
TABLE,
{ pk: 'u7', sk: 'once', v: 1 },
{
condition: absent,
},
);
await expect(
client.put(
TABLE,
{ pk: 'u7', sk: 'once', v: 2 },
{
condition: absent,
},
),
).rejects.toMatchObject({ name: 'ConditionalCheckFailedException' });
expect(
(await client.get(TABLE, { pk: 'u7', sk: 'once' })).Item?.v,
).toBe(1);
await client.put(
TABLE,
{ pk: 'u7', sk: 'once', v: 3 },
{
condition: {
expression: '#v = :held',
names: { '#v': 'v' },
values: { ':held': 1 },
},
},
);
expect(
(await client.get(TABLE, { pk: 'u7', sk: 'once' })).Item?.v,
).toBe(3);
});
it('fetches keys from several tables in one batch', async () => {
await client.put(TABLE, { pk: 'b1', sk: 'a', v: 1 });
await client.put(TABLE, { pk: 'b1', sk: 'b', v: 2 });
@@ -364,6 +451,59 @@ describe('DDBClient — expired item sweep', () => {
expect((await client.get(table, { pk: 'fresh' })).Item).toBeDefined();
expect((await client.get(table, { pk: 'eternal' })).Item).toBeDefined();
});
it('keeps an item whose ttl is 0, which never expires', async () => {
const client = new DDBClient(localConfig());
const table = 'ttl-zero-items';
const schema = {
TableName: table,
KeySchema: [{ AttributeName: 'pk', KeyType: 'HASH' as const }],
AttributeDefinitions: [
{ AttributeName: 'pk', AttributeType: 'S' as const },
],
ProvisionedThroughput: {
ReadCapacityUnits: 5,
WriteCapacityUnits: 5,
},
};
await client.createTableIfNotExists(schema, 'expireAt');
await awaitTable(client, table, { pk: 'ready' });
await client.put(table, { pk: 'zero', expireAt: 0 });
// Second call finds the table in use, so the sweep runs.
await client.createTableIfNotExists(schema, 'expireAt');
expect((await client.get(table, { pk: 'zero' })).Item).toBeDefined();
});
it('keeps an item whose ttl is not a number, which the sweep can never purge', async () => {
const client = new DDBClient(localConfig());
const table = 'ttl-nan-items';
const schema = {
TableName: table,
KeySchema: [{ AttributeName: 'pk', KeyType: 'HASH' as const }],
AttributeDefinitions: [
{ AttributeName: 'pk', AttributeType: 'S' as const },
],
ProvisionedThroughput: {
ReadCapacityUnits: 5,
WriteCapacityUnits: 5,
},
};
await client.createTableIfNotExists(schema, 'expireAt');
await awaitTable(client, table, { pk: 'ready' });
const past = Math.floor(Date.now() / 1000) - 60;
await client.put(table, { pk: 'stringTtl', expireAt: String(past) });
// Second call finds the table in use, so the sweep runs.
await client.createTableIfNotExists(schema, 'expireAt');
expect(
(await client.get(table, { pk: 'stringTtl' })).Item,
).toBeDefined();
});
});
describe('DDBClient — expired item sweep on an empty table', () => {
@@ -402,6 +542,19 @@ describe('DDBClient — credentialled configuration', () => {
).toThrow('requires both `access_key` and `secret_key`');
});
it('accepts the credential chain in place of keys', async () => {
const client = new DDBClient({
port: 0,
extensions: [],
dynamo: {
aws: { useCredentialChain: true, region: 'us-west-2' },
endpoint: 'http://127.0.0.1:1',
},
} as unknown as IConfig);
await expect(client.recreateClient()).resolves.toBeUndefined();
});
it('skips table creation on a managed deployment', async () => {
const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
const client = new DDBClient({
@@ -674,3 +827,391 @@ describe('DDBClient — unprocessed batch keys', () => {
).resolves.toEqual({ Responses: {}, ConsumedCapacity: [] });
});
});
// Answers by operation (X-Amz-Target) and records how many requests overlap, so
// chunk concurrency and the TTL sweep's retries can be asserted.
describe('DDBClient — batch write chunks and TTL sweep', () => {
let server: Server;
let endpoint: string;
let handlers: Record<string, (body: Record<string, unknown>) => unknown>;
let calls: string[];
let inFlight = 0;
let peakInFlight = 0;
beforeAll(async () => {
server = createServer((req, res) => {
let body = '';
req.on('data', (chunk) => {
body += chunk;
});
req.on('end', () => {
const op = String(req.headers['x-amz-target']).split('.')[1];
calls.push(op);
inFlight += 1;
peakInFlight = Math.max(peakInFlight, inFlight);
setTimeout(() => {
inFlight -= 1;
const reply = handlers[op]?.(JSON.parse(body)) ?? {};
const { status = 200, ...payload } = reply as {
status?: number;
};
res.writeHead(status, {
'content-type': 'application/x-amz-json-1.0',
});
res.end(JSON.stringify(payload));
}, 20);
});
});
await new Promise<void>((resolve) =>
server.listen(0, '127.0.0.1', resolve),
);
endpoint = `http://127.0.0.1:${(server.address() as AddressInfo).port}`;
});
afterAll(async () => {
await new Promise<void>((resolve) => server.close(() => resolve()));
});
beforeEach(() => {
calls = [];
inFlight = 0;
peakInFlight = 0;
});
const stubClient = (extra: Record<string, unknown> = {}) =>
new DDBClient({
port: 0,
extensions: [],
dynamo: {
aws: { access_key: 'a', secret_key: 'b', region: 'us-west-2' },
endpoint,
...extra,
},
} as unknown as IConfig);
it('writes chunks concurrently, a bounded number at a time', async () => {
handlers = { BatchWriteItem: () => ({ UnprocessedItems: {} }) };
await stubClient().batchPut(
Array.from({ length: 200 }, (_, i) => ({
table: TABLE,
item: { pk: `item-${i}` },
})),
);
expect(calls).toHaveLength(8);
expect(peakInFlight).toBeGreaterThan(1);
expect(peakInFlight).toBeLessThanOrEqual(4);
});
it('stops starting chunks once one fails', async () => {
handlers = {
BatchWriteItem: () => ({
status: 400,
__type: 'com.amazon.coral.validate#ValidationException',
message: 'bad item',
}),
};
await expect(
stubClient().batchPut(
Array.from({ length: 250 }, (_, i) => ({
table: TABLE,
item: { pk: `item-${i}` },
})),
),
).rejects.toThrow('bad item');
// The first wave fails; none of the six chunks queued behind it run.
expect(calls.length).toBeLessThanOrEqual(4);
});
it('lets chunks already in flight finish before rejecting', async () => {
let replies = 0;
handlers = {
BatchWriteItem: (body) => {
replies += 1;
if (replies === 1) {
return {
status: 400,
__type: 'com.amazon.coral.validate#ValidationException',
message: 'bad item',
};
}
// The rest need one retry, so they are still writing when the
// failure comes back.
return replies <= 4
? { UnprocessedItems: body.RequestItems }
: { UnprocessedItems: {} };
},
};
await expect(
stubClient().batchPut(
Array.from({ length: 100 }, (_, i) => ({
table: TABLE,
item: { pk: `item-${i}` },
})),
),
).rejects.toThrow('bad item');
const sentBeforeReject = calls.length;
await new Promise((resolve) => setTimeout(resolve, 200));
// Nothing is written after the caller has seen the error.
expect(calls.length).toBe(sentBeforeReject);
expect(inFlight).toBe(0);
});
it('retries the unprocessed deletes of the TTL sweep', async () => {
const expired = [{ pk: { S: 'old' } }];
let batchWrites = 0;
handlers = {
CreateTable: () => ({
status: 400,
__type: 'com.amazonaws.dynamodb.v20120810#ResourceInUseException',
message: 'Table already exists',
}),
Scan: () => ({ Items: expired }),
BatchWriteItem: (body) => {
batchWrites += 1;
return batchWrites === 1
? { UnprocessedItems: body.RequestItems }
: { UnprocessedItems: {} };
},
};
await stubClient({ bootstrapTables: true }).createTableIfNotExists(
{
TableName: TABLE,
KeySchema: [{ AttributeName: 'pk', KeyType: 'HASH' }],
AttributeDefinitions: [
{ AttributeName: 'pk', AttributeType: 'S' },
],
BillingMode: 'PAY_PER_REQUEST',
},
'ttl',
);
expect(calls).toEqual([
'CreateTable',
'Scan',
'BatchWriteItem',
'BatchWriteItem',
]);
});
});
// Throttles are a service-side signal the emulator never produces, so these
// drive the client against a stub answering in DynamoDB's error shape.
describe('DDBClient — throttle logging', () => {
let server: Server;
let endpoint: string;
let responses: { status: number; body: unknown }[];
let warn: MockInstance<typeof console.warn>;
beforeAll(async () => {
server = createServer((req, res) => {
req.resume();
req.on('end', () => {
const next = responses.shift() ?? { status: 200, body: {} };
res.writeHead(next.status, {
'content-type': 'application/x-amz-json-1.0',
});
res.end(JSON.stringify(next.body));
});
});
await new Promise<void>((resolve) =>
server.listen(0, '127.0.0.1', resolve),
);
endpoint = `http://127.0.0.1:${(server.address() as AddressInfo).port}`;
});
afterAll(async () => {
await new Promise<void>((resolve) => server.close(() => resolve()));
});
beforeEach(() => {
warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
});
afterEach(() => {
vi.restoreAllMocks();
});
const stubClient = () =>
new DDBClient({
port: 0,
extensions: [],
dynamo: {
aws: {
access_key: 'a',
secret_key: 'b',
region: 'eu-central-1',
},
endpoint,
},
} as unknown as IConfig);
const throttled = (reason = 'TableWriteKeyRangeThroughputExceeded') => ({
status: 400,
body: {
__type: 'com.amazonaws.dynamodb.v20120810#ProvisionedThroughputExceededException',
message:
'The level of configured provisioned throughput for the table was exceeded.',
ThrottlingReasons: [
{
reason,
resource: `arn:aws:dynamodb:eu-central-1:0:table/${TABLE}`,
},
],
},
});
const ok = (body: unknown = {}) => ({ status: 200, body });
const ddbWarnings = () =>
warn.mock.calls
.map(([line]) => String(line))
.filter((line) => line.startsWith('[ddb] '));
it('logs the key of a throttled update that succeeds on retry', async () => {
responses = [
throttled(),
ok({
Attributes: {
pk: { S: 'hot' },
sk: { S: 'counter' },
v: { N: '1' },
},
}),
];
const result = await stubClient().update(
TABLE,
{ pk: 'hot', sk: 'counter' },
'SET v = :v',
{ ':v': 1 },
);
expect(result.Attributes?.v).toBe(1);
expect(ddbWarnings()).toEqual([
'[ddb] throttled UpdateItem kv-items pk=hot sk=counter in eu-central-1: TableWriteKeyRangeThroughputExceeded',
]);
}, 15_000);
it('logs only the key attributes of a throttled put', async () => {
const client = stubClient();
// On a managed deployment this creates nothing but learns the key schema.
await client.createTableIfNotExists(tableSchema);
responses = [throttled(), ok()];
await client.put(TABLE, {
pk: 'user-1',
sk: 'settings',
secret: 'do-not-log',
});
expect(ddbWarnings()).toEqual([
'[ddb] throttled PutItem kv-items pk=user-1 sk=settings in eu-central-1: TableWriteKeyRangeThroughputExceeded',
]);
expect(warn.mock.calls.flat().join('\n')).not.toContain('do-not-log');
}, 15_000);
it('logs a hot key once a minute and counts the throttles in between', async () => {
const now = vi.spyOn(Date, 'now').mockReturnValue(1_000_000);
const client = stubClient();
const read = 'TableReadKeyRangeThroughputExceeded';
responses = [throttled(read), ok(), throttled(read), ok()];
await client.get(TABLE, { pk: 'burst', sk: 'a' });
await client.get(TABLE, { pk: 'burst', sk: 'a' });
now.mockReturnValue(1_000_000 + 61_000);
responses = [throttled(read), ok()];
await client.get(TABLE, { pk: 'burst', sk: 'a' });
expect(ddbWarnings()).toEqual([
`[ddb] throttled GetItem kv-items pk=burst sk=a in eu-central-1: ${read}`,
`[ddb] throttled GetItem kv-items pk=burst sk=a in eu-central-1: ${read} (+1 since last logged)`,
]);
}, 15_000);
it('names the partition a throttled query read', async () => {
responses = [throttled('IndexReadKeyRangeThroughputExceeded'), ok()];
await stubClient().query(TABLE, { kind: 'note' }, 0, undefined, GSI);
expect(ddbWarnings()).toEqual([
'[ddb] throttled Query kv-items index=by-kind kind=note in eu-central-1: IndexReadKeyRangeThroughputExceeded',
]);
}, 15_000);
it('names up to five keys of a throttled batch', async () => {
responses = [
throttled('TableReadKeyRangeThroughputExceeded'),
ok({ Responses: {}, UnprocessedKeys: {} }),
];
await stubClient().batchGet(
Array.from({ length: 7 }, (_, index) => ({
table: TABLE,
items: { pk: `batch-${index}`, sk: 's' },
})),
);
expect(ddbWarnings()).toEqual([
'[ddb] throttled BatchGetItem kv-items pk=batch-0 sk=s, pk=batch-1 sk=s, pk=batch-2 sk=s, pk=batch-3 sk=s, pk=batch-4 sk=s (+2 more) in eu-central-1: TableReadKeyRangeThroughputExceeded',
]);
}, 15_000);
it('logs batch items DynamoDB hands back unprocessed', async () => {
const client = stubClient();
await client.createTableIfNotExists(tableSchema);
responses = [
ok({
UnprocessedItems: {
[TABLE]: [
{
PutRequest: {
Item: {
pk: { S: 'left' },
sk: { S: 'over' },
v: { N: '1' },
},
},
},
],
},
}),
ok({ UnprocessedItems: {} }),
];
await client.batchPut([
{ table: TABLE, item: { pk: 'left', sk: 'over', v: 1 } },
{ table: TABLE, item: { pk: 'done', sk: 'fine', v: 2 } },
]);
expect(ddbWarnings()).toEqual([
'[ddb] unprocessed BatchWriteItem kv-items pk=left sk=over in eu-central-1',
]);
});
it('does not log errors that are not throttles', async () => {
responses = [
{
status: 400,
body: {
__type: 'com.amazonaws.dynamodb.v20120810#ConditionalCheckFailedException',
message: 'The conditional request failed',
},
},
];
await expect(
stubClient().del(
TABLE,
{ pk: 'guarded', sk: 'a' },
{ condition: { expression: 'attribute_exists(pk)' } },
),
).rejects.toMatchObject({ name: 'ConditionalCheckFailedException' });
expect(ddbWarnings()).toEqual([]);
});
});
+253 -223
View File
@@ -43,6 +43,12 @@ import { once } from 'node:events';
import { Agent as httpsAgent } from 'node:https';
import { HttpError } from '../../core/http';
import type { IConfig, IDynamoConfig } from '../../types';
import {
awsCredentialsFrom,
type AwsCredentials,
} from '../../util/awsCredentials.js';
import { BoundedTtlMap } from '../../util/boundedTtlMap.js';
import { runWithConcurrencyLimitSettled } from '../../util/concurrency.js';
import { Span } from '../../util/span.js';
import { PuterClient } from '../types';
import {
@@ -50,14 +56,15 @@ import {
isRepairableMarshallError,
repairForMarshall,
} from './marshallRepair.js';
import { attachThrottleLogging } from './throttleLog.js';
const LOCAL_DYNAMO_MEMORY_PREFIX = ':memory:';
const localDynaliteEndpointPromises = new Map<string, Promise<string>>();
const MAX_BATCH_WRITE_ITEMS = 25;
const MAX_BATCH_WRITE_RETRIES = 8;
const BATCH_WRITE_RETRY_BASE_MS = 25;
const MAX_BATCH_GET_RETRIES = 8;
const BATCH_GET_RETRY_BASE_MS = 25;
// Chunks of one batch write in flight at once.
const BATCH_WRITE_CONCURRENCY = 4;
const MAX_BATCH_RETRIES = 8;
const BATCH_RETRY_BASE_MS = 25;
// In-memory mode gives each DDBClient its own unique key, which keys
// into a fresh dynalite server. This is what test parallelism needs:
@@ -117,21 +124,45 @@ const chunkValues = <T>(values: T[], size: number): T[][] => {
return chunks;
};
type BatchWriteItems = NonNullable<BatchWriteCommandInput['RequestItems']>;
type WriteRequest = BatchWriteItems[string][number];
/** Consumed capacity summed per table across the requests of one batch. */
class ConsumedCapacityTally {
#byTable = new Map<string, number>();
add(entries: { TableName?: string; CapacityUnits?: number }[] = []) {
for (const { TableName, CapacityUnits } of entries) {
if (!TableName) continue;
this.#byTable.set(
TableName,
(this.#byTable.get(TableName) ?? 0) +
Number(CapacityUnits ?? 0),
);
}
}
entries() {
return Array.from(this.#byTable, ([TableName, CapacityUnits]) => ({
TableName,
CapacityUnits,
}));
}
}
const sleep = async (ms: number) => {
await new Promise((resolve) => setTimeout(resolve, ms));
};
// One caller writing an out-of-range number usually keeps doing it, so the
// warning is per-target and throttled rather than one line per write.
const REPAIR_WARNING_INTERVAL_MS = 60_000;
const lastRepairWarningAt = new Map<string, number>();
const repairWarnings = new BoundedTtlMap<string, true>({
maxEntries: 1_000,
ttlMs: 60_000,
});
const warnRepaired = (target: string, message: string): void => {
const now = Date.now();
const lastWarnedAt = lastRepairWarningAt.get(target) ?? 0;
if (now - lastWarnedAt < REPAIR_WARNING_INTERVAL_MS) return;
lastRepairWarningAt.set(target, now);
if (!repairWarnings.shouldEmit(target)) return;
console.warn(
`[ddb] clamped an out-of-range value in ${target}: ${message}`,
);
@@ -172,6 +203,7 @@ export class DDBClient extends PuterClient {
// key per instance so parallel clients don't share state, but
// `recreateClient()` reuses this same key and so reuses the server.
#localPathKey: string;
#keyNamesByTable = new Map<string, string[]>();
constructor(config: IConfig) {
super(config);
@@ -220,7 +252,19 @@ export class DDBClient extends PuterClient {
}
@Span('ddb.put', (table: string) => ({ 'db.table': table }))
async put<T extends Record<string, unknown>>(table: string, item: T) {
async put<T extends Record<string, unknown>>(
table: string,
item: T,
opts?: {
/** Refused (`ConditionalCheckFailedException`) unless it holds. */
condition?: {
expression: string;
names?: Record<string, string>;
values?: Record<string, unknown>;
};
},
) {
const condition = opts?.condition;
const client = await this.#getDocumentClient();
return sendRepairingValues(
item,
@@ -230,6 +274,23 @@ export class DDBClient extends PuterClient {
TableName: table,
Item: itemToWrite,
ReturnConsumedCapacity: 'TOTAL',
...(condition
? {
ConditionExpression: condition.expression,
...(condition.names
? {
ExpressionAttributeNames:
condition.names,
}
: {}),
...(condition.values
? {
ExpressionAttributeValues:
condition.values,
}
: {}),
}
: {}),
}),
),
() => `a put to ${table}`,
@@ -255,9 +316,7 @@ export class DDBClient extends PuterClient {
{} as Record<string, Record<string, unknown>[]>,
);
let requestItems: BatchGetCommandInput['RequestItems'] = Object.entries(
allRequestItemsPerTable,
).reduce(
const requestItems = Object.entries(allRequestItemsPerTable).reduce(
(acc, [table, keyList]) => {
acc[table] = {
Keys: keyList,
@@ -270,53 +329,30 @@ export class DDBClient extends PuterClient {
const client = await this.#getDocumentClient();
const responsesByTable = new Map<string, Record<string, unknown>[]>();
const consumedCapacityByTable = new Map<string, number>();
const capacity = new ConsumedCapacityTally();
for (let attempt = 0; attempt <= MAX_BATCH_GET_RETRIES; attempt++) {
if (Object.keys(requestItems).length === 0) break;
const response = await client.send(
new BatchGetCommand({
RequestItems: requestItems,
ReturnConsumedCapacity: 'TOTAL',
}),
);
for (const [table, items] of Object.entries(
response.Responses ?? {},
)) {
const existing = responsesByTable.get(table) ?? [];
existing.push(...items);
responsesByTable.set(table, existing);
}
for (const entry of response.ConsumedCapacity ?? []) {
if (!entry.TableName) continue;
consumedCapacityByTable.set(
entry.TableName,
(consumedCapacityByTable.get(entry.TableName) ?? 0) +
Number(entry.CapacityUnits ?? 0),
const drained = await this.#sendWithUnprocessedRetry(
requestItems,
async (pending) => {
const response = await client.send(
new BatchGetCommand({
RequestItems: pending,
ReturnConsumedCapacity: 'TOTAL',
}),
);
}
for (const [table, items] of Object.entries(
response.Responses ?? {},
)) {
const existing = responsesByTable.get(table) ?? [];
existing.push(...items);
responsesByTable.set(table, existing);
}
capacity.add(response.ConsumedCapacity);
return response.UnprocessedKeys as typeof pending | undefined;
},
);
const unprocessedKeys = response.UnprocessedKeys ?? {};
if (Object.keys(unprocessedKeys).length === 0) {
requestItems = {};
break;
}
requestItems = unprocessedKeys as NonNullable<
BatchGetCommandInput['RequestItems']
>;
if (attempt < MAX_BATCH_GET_RETRIES) {
const delayMs = Math.min(
1000,
BATCH_GET_RETRY_BASE_MS * 2 ** attempt,
);
await sleep(delayMs);
}
}
if (Object.keys(requestItems).length > 0) {
if (!drained) {
throw new HttpError(
400,
'Failed to batch get all items from DynamoDB',
@@ -326,12 +362,7 @@ export class DDBClient extends PuterClient {
return {
Responses: Object.fromEntries(responsesByTable),
ConsumedCapacity: Array.from(consumedCapacityByTable.entries()).map(
([TableName, CapacityUnits]) => ({
TableName,
CapacityUnits,
}),
),
ConsumedCapacity: capacity.entries(),
};
}
@@ -359,129 +390,110 @@ export class DDBClient extends PuterClient {
);
}
// Shared BatchWriteItem plumbing for batchPut/batchDel: 25-item chunks,
// UnprocessedItems retried with capped exponential backoff, consumed
// capacity accumulated per table across every request.
async #batchWrite(
params: {
table: string;
request: NonNullable<
BatchWriteCommandInput['RequestItems']
>[string][number];
}[],
) {
const consumedCapacityByTable = new Map<string, number>();
// Shared BatchWriteItem plumbing for batchPut/batchDel and the TTL sweep:
// 25-item chunks a few at a time, UnprocessedItems retried, consumed
// capacity summed per table across every request.
async #batchWrite(params: { table: string; request: WriteRequest }[]) {
if (params.length === 0) {
return { ConsumedCapacity: [] };
}
const accumulateConsumedCapacity = (
consumedCapacityEntries:
| Array<{ TableName?: string; CapacityUnits?: number }>
| undefined,
) => {
if (!consumedCapacityEntries) {
return;
}
for (const consumedCapacityEntry of consumedCapacityEntries) {
const table = consumedCapacityEntry.TableName;
if (!table) {
continue;
}
const existingUsage = consumedCapacityByTable.get(table) ?? 0;
consumedCapacityByTable.set(
table,
existingUsage +
Number(consumedCapacityEntry.CapacityUnits ?? 0),
);
}
};
const client = await this.#getDocumentClient();
const chunks = chunkValues(params, MAX_BATCH_WRITE_ITEMS);
const capacity = new ConsumedCapacityTally();
// Once a chunk fails, the rest aren't started. Those in flight settle
// before the error is thrown, so no write lands after the caller's
// cleanup.
let failed = false;
for (const chunk of chunks) {
let requestItems = chunk.reduce(
(acc, curr) => {
const tableRequests = acc[curr.table] ?? [];
tableRequests.push(curr.request);
acc[curr.table] = tableRequests;
return acc;
},
{} as NonNullable<BatchWriteCommandInput['RequestItems']>,
);
for (
let attempt = 0;
attempt <= MAX_BATCH_WRITE_RETRIES;
attempt++
) {
if (Object.keys(requestItems).length === 0) {
break;
const outcomes = await runWithConcurrencyLimitSettled(
chunkValues(params, MAX_BATCH_WRITE_ITEMS),
BATCH_WRITE_CONCURRENCY,
async (chunk) => {
if (failed) return;
const requestItems: BatchWriteItems = {};
for (const { table, request } of chunk) {
(requestItems[table] ??= []).push(request);
}
const response = await sendRepairingValues(
requestItems,
(itemsToWrite) =>
client.send(
new BatchWriteCommand({
RequestItems: itemsToWrite,
ReturnConsumedCapacity: 'TOTAL',
}),
),
() =>
`a batch write to ${Object.keys(requestItems).join(', ')}`,
);
accumulateConsumedCapacity(
response.ConsumedCapacity as
| Array<{ TableName?: string; CapacityUnits?: number }>
| undefined,
);
const unprocessedItems = response.UnprocessedItems ?? {};
if (Object.keys(unprocessedItems).length === 0) {
requestItems = {};
break;
}
requestItems = unprocessedItems as NonNullable<
BatchWriteCommandInput['RequestItems']
>;
if (attempt < MAX_BATCH_WRITE_RETRIES) {
const delayMs = Math.min(
1000,
BATCH_WRITE_RETRY_BASE_MS * 2 ** attempt,
try {
const drained = await this.#sendWithUnprocessedRetry(
requestItems,
async (pending) => {
const response = await sendRepairingValues(
pending,
(itemsToWrite) =>
client.send(
new BatchWriteCommand({
RequestItems: itemsToWrite,
ReturnConsumedCapacity: 'TOTAL',
}),
),
() =>
`a batch write to ${Object.keys(pending).join(', ')}`,
);
capacity.add(response.ConsumedCapacity);
return response.UnprocessedItems as
BatchWriteItems | undefined;
},
);
await sleep(delayMs);
if (!drained) {
throw new HttpError(
400,
'Failed to batch write all items to DynamoDB',
{ legacyCode: 'bad_request' },
);
}
} catch (error) {
failed = true;
throw error;
}
}
},
);
const rejected = outcomes.find(
(outcome): outcome is PromiseRejectedResult =>
outcome.status === 'rejected',
);
if (rejected) throw rejected.reason;
if (Object.keys(requestItems).length > 0) {
throw new HttpError(
400,
'Failed to batch write all items to DynamoDB',
{ legacyCode: 'bad_request' },
);
return { ConsumedCapacity: capacity.entries() };
}
/**
* Send a batch request and resend whatever comes back unprocessed, with
* capped exponential backoff. `send` returns the unprocessed remainder.
* Resolves false when items are still left after the last retry.
*/
async #sendWithUnprocessedRetry<TItems extends object>(
requestItems: TItems,
send: (pending: TItems) => Promise<TItems | undefined>,
): Promise<boolean> {
let pending: TItems | undefined = requestItems;
for (let attempt = 0; attempt <= MAX_BATCH_RETRIES; attempt++) {
if (!pending || Object.keys(pending).length === 0) return true;
pending = await send(pending);
if (
attempt < MAX_BATCH_RETRIES &&
pending &&
Object.keys(pending).length > 0
) {
await sleep(Math.min(1000, BATCH_RETRY_BASE_MS * 2 ** attempt));
}
}
return {
ConsumedCapacity: Array.from(consumedCapacityByTable.entries()).map(
([TableName, CapacityUnits]) => ({
TableName,
CapacityUnits,
}),
),
};
return !pending || Object.keys(pending).length === 0;
}
@Span('ddb.del', (table: string) => ({ 'db.table': table }))
async del<T extends Record<string, unknown>>(
table: string,
key: T,
opts?: { returnOld?: boolean },
opts?: {
returnOld?: boolean;
/** Refused (`ConditionalCheckFailedException`) unless it holds. */
condition?: {
expression: string;
names?: Record<string, string>;
values?: Record<string, unknown>;
};
},
) {
const command = new DeleteCommand({
TableName: table,
@@ -490,6 +502,20 @@ export class DDBClient extends PuterClient {
// ALL_OLD makes the delete an atomic claim: exactly one caller
// gets the attributes back.
...(opts?.returnOld ? { ReturnValues: 'ALL_OLD' as const } : {}),
...(opts?.condition
? {
ConditionExpression: opts.condition.expression,
...(opts.condition.names
? { ExpressionAttributeNames: opts.condition.names }
: {}),
...(opts.condition.values
? {
ExpressionAttributeValues:
opts.condition.values,
}
: {}),
}
: {}),
});
const client = await this.#getDocumentClient();
@@ -611,10 +637,25 @@ export class DDBClient extends PuterClient {
);
}
/** Lets throttle logs name a put's key without logging the item. */
registerKeySchema(
table: string,
keySchema: NonNullable<CreateTableCommandInput['KeySchema']>,
) {
this.#keyNamesByTable.set(
table,
keySchema.map((key) => key.AttributeName!),
);
}
async createTableIfNotExists(
params: CreateTableCommandInput,
ttlAttribute?: string,
) {
if (params.TableName && params.KeySchema) {
this.registerKeySchema(params.TableName, params.KeySchema);
}
// Real-AWS deployments provision tables externally (Terraform / IaC),
// so we no-op there by default. Self-hosters pointing at
// dynamodb-local opt in via `dynamo.bootstrapTables: true`.
@@ -680,39 +721,17 @@ export class DDBClient extends PuterClient {
}
#bindAwsClient() {
const accessKeyId = this.#ddbConfig.aws?.access_key;
const secretAccessKey = this.#ddbConfig.aws?.secret_key;
if (!accessKeyId || !secretAccessKey) {
const credentials = awsCredentialsFrom(this.#ddbConfig.aws);
if (!credentials) {
throw new Error(
'DynamoDB aws config requires both `access_key` and `secret_key`',
'DynamoDB aws config requires both `access_key` and `secret_key`, or `useCredentialChain`',
);
}
const ddbClient = new DynamoDBClient({
credentials: {
accessKeyId,
secretAccessKey,
},
maxAttempts: 3,
requestHandler: new NodeHttpHandler({
connectionTimeout: 5000,
requestTimeout: 5000,
httpsAgent: new httpsAgent({ keepAlive: true }),
}),
...(this.#ddbConfig.endpoint
? { endpoint: this.#ddbConfig.endpoint }
: {}),
this.#bindClient({
credentials,
region: this.#ddbConfig.aws?.region || 'us-west-2',
});
this.#documentClient = DynamoDBDocumentClient.from(ddbClient, {
marshallOptions: {
removeUndefinedValues: true,
},
unmarshallOptions: {
wrapNumbers: clampStoredNumber,
},
endpoint: this.#ddbConfig.endpoint,
});
}
@@ -720,21 +739,36 @@ export class DDBClient extends PuterClient {
const endpoint = await getOrCreateLocalDynaliteEndpoint(
this.#localPathKey,
);
this.#bindClient({
credentials: { accessKeyId: 'fake', secretAccessKey: 'fake' },
region: 'us-west-2',
endpoint,
});
}
#bindClient({
credentials,
region,
endpoint,
}: {
credentials: AwsCredentials;
region: string;
endpoint?: string;
}) {
const ddbClient = new DynamoDBClient({
credentials: {
accessKeyId: 'fake',
secretAccessKey: 'fake',
},
credentials,
maxAttempts: 3,
requestHandler: new NodeHttpHandler({
connectionTimeout: 5000,
requestTimeout: 5000,
httpsAgent: new httpsAgent({ keepAlive: true }),
}),
endpoint,
region: 'us-west-2',
...(endpoint ? { endpoint } : {}),
region,
});
attachThrottleLogging(ddbClient, region, (table) =>
this.#keyNamesByTable.get(table),
);
this.#documentClient = DynamoDBDocumentClient.from(ddbClient, {
marshallOptions: {
@@ -761,14 +795,15 @@ export class DDBClient extends PuterClient {
const scan = await client.send(
new ScanCommand({
TableName: table,
FilterExpression: '#ttl < :now',
// 0 means no expiry, so it must never sweep.
FilterExpression: '#ttl < :now AND #ttl <> :none',
ExpressionAttributeNames: {
'#ttl': ttlAttribute,
...Object.fromEntries(
keyNames.map((key) => [`#k_${key}`, key]),
),
},
ExpressionAttributeValues: { ':now': now },
ExpressionAttributeValues: { ':now': now, ':none': 0 },
ProjectionExpression: keyNames
.map((key) => `#k_${key}`)
.join(', '),
@@ -779,27 +814,22 @@ export class DDBClient extends PuterClient {
);
lastEvaluatedKey = scan.LastEvaluatedKey as
| Record<string, unknown>
| undefined;
Record<string, unknown> | undefined;
const items = scan.Items;
if (!items || items.length === 0) continue;
const chunks = chunkValues(items, MAX_BATCH_WRITE_ITEMS);
for (const chunk of chunks) {
await client.send(
new BatchWriteCommand({
RequestItems: {
[table]: chunk.map((item) => ({
DeleteRequest: {
Key: Object.fromEntries(
keyNames.map((key) => [key, item[key]]),
),
},
})),
await this.#batchWrite(
items.map((item) => ({
table,
request: {
DeleteRequest: {
Key: Object.fromEntries(
keyNames.map((key) => [key, item[key]]),
),
},
}),
);
}
},
})),
);
} while (lastEvaluatedKey);
}
}
+245
View File
@@ -0,0 +1,245 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import type { DynamoDBClient } from '@aws-sdk/client-dynamodb';
import { BoundedTtlMap } from '../../util/boundedTtlMap.js';
const THROTTLE_ERROR_NAMES = new Set([
'ProvisionedThroughputExceededException',
'ThrottlingException',
'RequestLimitExceeded',
]);
// A hot key throttles in bursts, so each target logs at most once a minute and
// the next line says how many were folded into it.
const LOG_INTERVAL_MS = 60_000;
const MAX_TRACKED_TARGETS = 1_000;
const MAX_KEYS_PER_LINE = 5;
const MAX_VALUE_LENGTH = 120;
type AnyRecord = Record<string, unknown>;
type KeyNamesFor = (table: string) => string[] | undefined;
interface Target {
table: string;
keys: string;
}
// No TTL: an entry past its interval still carries the count to report.
const lastLogged = new BoundedTtlMap<string, { at: number; folded: number }>({
maxEntries: MAX_TRACKED_TARGETS,
});
const logOnce = (target: string, detail?: string): void => {
const now = Date.now();
const previous = lastLogged.get(target);
if (previous && now - previous.at < LOG_INTERVAL_MS) {
previous.folded += 1;
return;
}
lastLogged.set(target, { at: now, folded: 0 });
const folded = previous?.folded
? ` (+${previous.folded} since last logged)`
: '';
console.warn(`[ddb] ${target}${detail ? `: ${detail}` : ''}${folded}`);
};
// Inputs reach this middleware marshalled and outputs arrive unmarshalled, so
// a key value can be either shape.
const formatValue = (value: unknown): string => {
let text: string;
if (value && typeof value === 'object') {
const attr = value as { S?: string; N?: string; B?: unknown };
text =
attr.S ??
attr.N ??
(attr.B !== undefined ? '<binary>' : JSON.stringify(value));
} else {
text = String(value);
}
return text.length > MAX_VALUE_LENGTH
? `${text.slice(0, MAX_VALUE_LENGTH)}...`
: text;
};
const formatKey = (key: unknown, names?: string[]): string => {
if (!key || typeof key !== 'object') return '<no key>';
const record = key as AnyRecord;
const parts = (names ?? Object.keys(record))
.filter((name) => name in record)
.map((name) => `${name}=${formatValue(record[name])}`);
return parts.length > 0 ? parts.join(' ') : '<no key>';
};
// A put carries the whole item; only its key attributes are logged, never values.
const formatItemKey = (
table: string,
item: unknown,
keyNamesFor: KeyNamesFor,
): string => {
const names = keyNamesFor(table);
return names ? formatKey(item, names) : '<key schema unknown>';
};
const formatList = (keys: string[]): string => {
const shown = keys.slice(0, MAX_KEYS_PER_LINE).join(', ');
const hidden = keys.length - MAX_KEYS_PER_LINE;
return hidden > 0 ? `${shown} (+${hidden} more)` : shown;
};
const formatQueryKey = (input: AnyRecord): string => {
const values = (input.ExpressionAttributeValues ?? {}) as AnyRecord;
const placeholders =
String(input.KeyConditionExpression ?? '').match(/:[\w-]+/g) ?? [];
const parts = placeholders.map(
(placeholder) =>
`${placeholder.slice(1)}=${formatValue(values[placeholder])}`,
);
const index = input.IndexName ? `index=${input.IndexName} ` : '';
return index + (parts.length > 0 ? parts.join(' ') : '<no key>');
};
const describeBatchWrite = (
requestItems: unknown,
keyNamesFor: KeyNamesFor,
): Target[] =>
Object.entries((requestItems ?? {}) as AnyRecord).map(
([table, requests]) => ({
table,
keys: formatList(
((requests ?? []) as AnyRecord[]).map((request) => {
const put = request.PutRequest as AnyRecord | undefined;
if (put) return formatItemKey(table, put.Item, keyNamesFor);
const del = request.DeleteRequest as AnyRecord | undefined;
return formatKey(del?.Key);
}),
),
}),
);
const describeBatchGet = (requestItems: unknown): Target[] =>
Object.entries((requestItems ?? {}) as AnyRecord).map(([table, entry]) => ({
table,
keys: formatList(
((entry as { Keys?: unknown[] })?.Keys ?? []).map((key) =>
formatKey(key),
),
),
}));
const describeRequest = (
commandName: string,
input: AnyRecord,
keyNamesFor: KeyNamesFor,
): Target[] => {
const table = String(input.TableName ?? '');
switch (commandName) {
case 'GetItemCommand':
case 'UpdateItemCommand':
case 'DeleteItemCommand':
return [{ table, keys: formatKey(input.Key) }];
case 'PutItemCommand':
return [
{ table, keys: formatItemKey(table, input.Item, keyNamesFor) },
];
case 'QueryCommand':
return [{ table, keys: formatQueryKey(input) }];
case 'BatchWriteItemCommand':
return describeBatchWrite(input.RequestItems, keyNamesFor);
case 'BatchGetItemCommand':
return describeBatchGet(input.RequestItems);
default:
return [{ table, keys: '<no key>' }];
}
};
const throttleReasons = (error: unknown): string => {
const err = error as {
name?: string;
ThrottlingReasons?: { reason?: string }[];
throttlingReasons?: { reason?: string }[];
};
const reasons = (err.ThrottlingReasons ?? err.throttlingReasons ?? [])
.map((entry) => entry.reason)
.filter(Boolean);
return reasons.length > 0 ? reasons.join(', ') : String(err.name);
};
/**
* Log the table and key of every throttled attempt, including the ones the SDK
* retries through, plus batch items DynamoDB hands back unprocessed. Sits
* inside the retry loop because the CloudWatch throttle metrics count attempts,
* and most throttled attempts succeed on retry.
*/
export const attachThrottleLogging = (
client: DynamoDBClient,
region: string,
keyNamesFor: KeyNamesFor,
): void => {
client.middlewareStack.addRelativeTo(
<TArgs extends { input: unknown }, TResult extends { output: unknown }>(
next: (args: TArgs) => Promise<TResult>,
context: { commandName?: string },
) =>
async (args: TArgs): Promise<TResult> => {
const commandName = context.commandName ?? '';
const operation = commandName.replace(/Command$/, '');
try {
const result = await next(args);
const output = result.output as {
UnprocessedItems?: unknown;
UnprocessedKeys?: unknown;
};
const leftovers = [
...describeBatchWrite(
output?.UnprocessedItems,
keyNamesFor,
),
...describeBatchGet(output?.UnprocessedKeys),
];
for (const { table, keys } of leftovers) {
logOnce(
`unprocessed ${operation} ${table} ${keys} in ${region}`,
);
}
return result;
} catch (error) {
if (THROTTLE_ERROR_NAMES.has((error as Error)?.name)) {
const targets = describeRequest(
commandName,
args.input as AnyRecord,
keyNamesFor,
);
for (const { table, keys } of targets) {
logOnce(
`throttled ${operation} ${table} ${keys} in ${region}`,
throttleReasons(error),
);
}
}
throw error;
}
},
{
name: 'throttleKeyLogger',
relation: 'after',
toMiddleware: 'retryMiddleware',
},
);
};
@@ -425,86 +425,3 @@ describe('EmailClient — share notification templates', () => {
expect(html).toContain('href="https://puter.test"');
});
});
describe('EmailClient.clean', () => {
const clean = (email: string) => startClient().clean(email);
it('strips subaddressing on ordinary domains', () => {
expect(clean('[email protected]')).toBe('[email protected]');
});
it('ignores dots and subaddressing for gmail', () => {
expect(clean('[email protected]')).toBe('[email protected]');
});
it('canonicalizes googlemail onto gmail', () => {
expect(clean('[email protected]')).toBe('[email protected]');
});
it('keeps yahoo subaddressing, which yahoo treats as distinct', () => {
expect(clean('[email protected]')).toBe('[email protected]');
});
it('ignores dots for icloud aliases', () => {
expect(clean('[email protected]')).toBe('[email protected]');
});
it('returns anything that is not an address unchanged', () => {
expect(clean('not-an-email')).toBe('not-an-email');
expect(clean('@nolocal.test')).toBe('@nolocal.test');
});
});
describe('EmailClient.validate', () => {
it('waves everything through in a dev environment', async () => {
const client = startClient({ env: 'dev' } as Partial<IConfig>);
await expect(client.validate('[email protected]')).resolves.toBe(
true,
);
});
it('rejects a blocked domain suffix', async () => {
const client = startClient({
blockedEmailDomains: ['blocked.test'],
} as unknown as Partial<IConfig>);
await expect(client.validate('[email protected]')).resolves.toBe(
false,
);
await expect(client.validate('[email protected]')).resolves.toBe(
true,
);
});
it('matches the blocklist against the cleaned address', async () => {
const client = startClient({
blockedEmailDomains: ['@gmail.com'],
} as unknown as Partial<IConfig>);
await expect(
client.validate('[email protected]'),
).resolves.toBe(false);
});
it('lets a registered validator veto an address', async () => {
const client = startClient();
const seen: string[] = [];
client.addValidator((email) => {
seen.push(email);
return !email.startsWith('spam');
});
await expect(client.validate('[email protected]')).resolves.toBe(
false,
);
await expect(client.validate('[email protected]')).resolves.toBe(true);
// Validators always see the canonical form.
expect(seen).toEqual(['[email protected]', '[email protected]']);
});
it('supports asynchronous validators', async () => {
const client = startClient();
client.addValidator(async (email) => email !== '[email protected]');
await expect(client.validate('[email protected]')).resolves.toBe(false);
await expect(client.validate('[email protected]')).resolves.toBe(true);
});
});
+2 -111
View File
@@ -71,54 +71,12 @@ export interface SendMailOptions {
headers?: Record<string, string>;
}
export type EmailValidator = (email: string) => Promise<boolean> | boolean;
interface CompiledTemplate {
subject: ReturnType<typeof template>;
html: ReturnType<typeof template>;
text?: ReturnType<typeof template>;
}
// -- Clean-email rules ------------------------------------------------
type CleanRule = (parts: { local: string; domain: string }) => {
local: string;
domain: string;
};
const CLEAN_RULES: Record<string, CleanRule> = {
dots_dont_matter: ({ local, domain }) => ({
local: local.replace(/\./g, ''),
domain,
}),
remove_subaddressing: ({ local, domain }) => ({
local: local.split('+')[0],
domain,
}),
};
const PROVIDER_RULES: Record<string, { apply: string[]; skip: string[] }> = {
gmail: { apply: ['dots_dont_matter'], skip: [] },
icloud: { apply: ['dots_dont_matter'], skip: [] },
yahoo: { apply: [], skip: ['remove_subaddressing'] },
};
const DOMAIN_TO_PROVIDER: Record<string, string> = {
'gmail.com': 'gmail',
'googlemail.com': 'gmail',
'yahoo.com': 'yahoo',
'yahoo.co.uk': 'yahoo',
'yahoo.ca': 'yahoo',
'yahoo.com.au': 'yahoo',
'icloud.com': 'icloud',
'me.com': 'icloud',
'mac.com': 'icloud',
};
const DOMAIN_ALIASES: Record<string, string> = {
'googlemail.com': 'gmail.com',
};
// -- EmailClient ------------------------------------------------------
/**
@@ -126,8 +84,8 @@ const DOMAIN_ALIASES: Record<string, string> = {
*
* - Template-based outbound mail (via `send`)
* - Raw nodemailer passthrough (via `sendRaw`)
* - Canonical-form normalization for dedup (via `clean`)
* - Policy + extensible validation (via `validate`)
*
* Address normalization and the domain blocklist live in `util/email.ts`.
*/
export class EmailClient extends PuterClient {
private transport: ReturnType<typeof nodemailer.createTransport> | null =
@@ -135,7 +93,6 @@ export class EmailClient extends PuterClient {
private compiledTemplates: Partial<
Record<EmailTemplateName, CompiledTemplate>
> = {};
private validators: EmailValidator[] = [];
constructor(config: IConfig) {
super(config);
@@ -218,72 +175,6 @@ export class EmailClient extends PuterClient {
});
}
// -- Public API: clean / validate ---------------------------------
/**
* Normalize an email to its canonical form for dedup comparisons. Applies
* provider-specific rules (e.g. Gmail ignores dots in the local part) plus
* generic subaddressing removal.
*/
clean(email: string): string {
let [local, domain] = email.split('@');
if (!local || !domain) return email;
if (DOMAIN_ALIASES[domain]) {
domain = DOMAIN_ALIASES[domain];
}
// Default: strip subaddressing on everything unless provider skips it
const ruleNames = new Set<string>(['remove_subaddressing']);
const provider = DOMAIN_TO_PROVIDER[domain];
const rules = provider ? PROVIDER_RULES[provider] : undefined;
if (rules) {
rules.apply.forEach((r) => ruleNames.add(r));
rules.skip.forEach((r) => ruleNames.delete(r));
}
let parts = { local, domain };
for (const name of ruleNames) {
parts = CLEAN_RULES[name](parts);
}
return `${parts.local}@${parts.domain}`;
}
/**
* Check whether an email is allowed to be used. Checks domain blocklist
* plus any registered validators (services can call `addValidator()` to
* register custom policy hooks).
*/
async validate(email: string): Promise<boolean> {
if (this.config.env === 'dev') return true;
const cleaned = this.clean(email);
const blocked = this.config.blockedEmailDomains;
if (Array.isArray(blocked)) {
for (const suffix of blocked) {
if (cleaned.endsWith(suffix)) return false;
}
}
for (const validator of this.validators) {
const ok = await validator(cleaned);
if (!ok) return false;
}
return true;
}
/**
* Register a custom validation hook. Services can call this during their
* startup to veto specific emails (e.g. a disposable-email service).
*/
addValidator(fn: EmailValidator): void {
this.validators.push(fn);
}
// -- Internals ----------------------------------------------------
private defaultFrom(): string {
+35 -4
View File
@@ -352,10 +352,10 @@ [email protected] immediately.
html: `
<p>Hi there,</p>
<p>{{team_name}} has created a Puter account for you.</p>
<p>Username: <b>{{username}}</b><br>
Temporary password: <b>{{temporary_password}}</b></p>
<p>You will be asked to choose your own password the first time you sign in.
This temporary one stops working then, and it expires on its own if unused.</p>
<p>Username: <b>{{username}}</b></p>
<p>{{team_name}} will give you a temporary password. You will be asked to choose
your own the first time you sign in. The temporary one stops working then, and it
expires on its own if unused.</p>
<p>What this means:</p>
<ul>
<li>This account belongs to {{team_name}}. They pay for it and can close it.</li>
@@ -364,6 +364,21 @@ This temporary one stops working then, and it expires on its own if unused.</p>
as you. Choose your own password promptly.</li>
</ul>
<p>Sincerely,</p>
<p>Puter</p>
`,
},
team_account_reissued: {
subject: 'A new temporary password for your {{team_name}} account',
html: `
<p>Hi there,</p>
<p>{{team_name}} has issued a new temporary password for your Puter account
<b>{{username}}</b>. Any temporary password you were given before this one has
stopped working.</p>
<p>{{team_name}} will give you the new one. You will be asked to choose your own
password the first time you sign in; the temporary one stops working then, and
it expires on its own if unused.</p>
<p>If you were not expecting this, ask {{team_name}} -- only they can issue it.</p>
<p>Sincerely,</p>
<p>Puter</p>
`,
},
@@ -404,6 +419,22 @@ will be asked to choose your own the next time you sign in.</p>
<p>If you did not expect this, the record of who reset it and when is in your
team activity, along with every sign-in to your account.</p>
<p>Sincerely,</p>
<p>Puter</p>
`,
},
team_2fa_reset: {
subject:
'Two-factor authentication was reset on your {{team_name}} account',
html: `
<p>Hi there,</p>
<p>An administrator of {{team_name}} reset two-factor authentication on your
Puter account <b>{{username}}</b>, so you can set it up again from a device you
still have. You have been signed out everywhere.</p>
<p>Your password is unchanged. If your team requires two-factor authentication,
you will be asked to enrol the next time you sign in.</p>
<p>If you did not expect this, the record of who reset it and when is in your
team activity, along with every sign-in to your account.</p>
<p>Sincerely,</p>
<p>Puter</p>
`,
},
@@ -1,3 +1,9 @@
import { trace } from '@opentelemetry/api';
import {
BasicTracerProvider,
InMemorySpanExporter,
SimpleSpanProcessor,
} from '@opentelemetry/sdk-trace-base';
import {
afterAll,
afterEach,
@@ -136,6 +142,30 @@ describe('EventClient', () => {
});
});
describe('repeat emits', () => {
it('reaches listeners added after the key was first emitted', () => {
target.emit(`${key}.a.b`, {}, {});
const wildcard = vi.fn();
const exact = vi.fn();
target.on(`${key}.a.*`, wildcard);
target.on(`${key}.a.b`, exact);
target.emit(`${key}.a.b`, {}, {});
expect(wildcard).toHaveBeenCalledTimes(1);
expect(exact).toHaveBeenCalledTimes(1);
expect(target.hasListeners(`${key}.a.b`)).toBe(true);
});
it('does not skip a listener when an earlier one unsubscribes mid-dispatch', () => {
const second = vi.fn();
const first = vi.fn(() => target.off(key, first));
target.on(key, first);
target.on(key, second);
target.emit(key, {}, {});
expect(first).toHaveBeenCalledTimes(1);
expect(second).toHaveBeenCalledTimes(1);
});
});
describe('emitAndWait', () => {
it('awaits async listeners before resolving', async () => {
let resolved = false;
@@ -183,6 +213,122 @@ describe('EventClient', () => {
await target.emitAndWait(`${key}.child`, {}, {});
expect(resolved).toBe(true);
});
describe('tracing', () => {
const exporter = new InMemorySpanExporter();
const provider = new BasicTracerProvider({
spanProcessors: [new SimpleSpanProcessor(exporter)],
});
const spansFor = (eventKey: string) =>
exporter
.getFinishedSpans()
.filter(
(s) =>
s.name === 'event.emitAndWait' &&
s.attributes['event.key'] === eventKey,
);
beforeAll(() => {
trace.setGlobalTracerProvider(provider);
});
afterAll(async () => {
await provider.shutdown();
trace.disable();
});
it('resolves without a span when nothing listens', async () => {
const meta: { listener_failed?: boolean } = {};
await expect(
target.emitAndWait(`${key}.child`, {}, meta),
).resolves.toBeUndefined();
expect(meta).toEqual({});
expect(spansFor(`${key}.child`)).toHaveLength(0);
});
it('spans the wait whenever a listener matches, wildcards included', async () => {
target.on(`${key}.*`, vi.fn());
await expect(
target.emitAndWait(`${key}.child`, {}, {}),
).resolves.toBeUndefined();
expect(spansFor(`${key}.child`)).toHaveLength(1);
extensionStore.events[`${key}.ext`] = [vi.fn()];
try {
await target.emitAndWait(`${key}.ext`, {}, {});
} finally {
delete extensionStore.events[`${key}.ext`];
}
expect(spansFor(`${key}.ext`)).toHaveLength(1);
});
});
});
describe('emitAndWaitConcurrently', () => {
it('runs every listener at once, wildcards included, and awaits them all', async () => {
let inFlight = 0;
let peak = 0;
const contribute =
(share: number) => async (_k: string, data: unknown) => {
inFlight++;
peak = Math.max(peak, inFlight);
await new Promise((r) => setTimeout(r, 10));
inFlight--;
(data as { total: number }).total += share;
};
target.on(`${key}.*`, contribute(1));
target.on(`${key}.child`, contribute(10));
target.on(`${key}.child`, contribute(100));
const data = { total: 0 };
await target.emitAndWaitConcurrently(`${key}.child`, data, {});
expect(peak).toBe(3);
expect(data.total).toBe(111);
});
it('flags a failed exact-key listener and still awaits the rest', async () => {
const errSpy = vi
.spyOn(console, 'error')
.mockImplementation(() => {});
let finished = false;
target.on(key, () => {
throw new Error('boom');
});
target.on(key, async () => {
await new Promise((r) => setTimeout(r, 5));
finished = true;
});
const meta: Record<string, unknown> = {};
await target.emitAndWaitConcurrently(key, {}, meta);
expect(finished).toBe(true);
expect(meta.listener_failed).toBe(true);
expect(errSpy).toHaveBeenCalledWith(
'Error in event listener for event',
key,
expect.any(Error),
);
errSpy.mockRestore();
});
it('leaves the flag alone when only a wildcard listener fails', async () => {
const errSpy = vi
.spyOn(console, 'error')
.mockImplementation(() => {});
target.on(`${key}.*`, () => {
throw new Error('observer');
});
const meta: Record<string, unknown> = {};
await target.emitAndWaitConcurrently(`${key}.child`, {}, meta);
expect(meta.listener_failed).toBeUndefined();
errSpy.mockRestore();
});
it('resolves at once with nobody listening', async () => {
await expect(
target.emitAndWaitConcurrently(`${key}.nobody`, {}, {}),
).resolves.toBeUndefined();
});
});
describe('off', () => {
+83 -34
View File
@@ -18,6 +18,7 @@
*/
import { extensionStore } from '../../extensions';
import { BoundedTtlMap } from '../../util/boundedTtlMap.js';
import { withSpan } from '../../util/span.js';
import { PuterClient } from '../types';
import {
@@ -28,8 +29,13 @@ import {
MatchingEvents,
} from './types';
const NO_LISTENERS: readonly EventListener[] = [];
export class EventClient extends PuterClient {
#eventListeners: Partial<Record<ListenKey, EventListener[]>> = {};
#matchKeyMemo = new BoundedTtlMap<string, readonly ListenKey[]>({
maxEntries: 1024,
});
onServerStart() {
this.emit('serverStart', {}, {});
@@ -63,19 +69,8 @@ export class EventClient extends PuterClient {
data: EventMap[T],
meta: EventMetadata,
) {
const parts = key.split('.');
for (let i = 0; i < parts.length; i++) {
const matchKey = (
i === parts.length - 1
? key
: `${parts.slice(0, i + 1).join('.')}.*`
) as ListenKey;
const extensionListeners = extensionStore.events[matchKey];
const listeners = (this.#eventListeners[matchKey] || []).concat(
extensionListeners || [],
);
if (!listeners) continue;
for (const listener of listeners) {
for (const matchKey of this.#matchKeys(key)) {
for (const listener of this.#listenersFor(matchKey)) {
this.#emitEvent(listener, key, data, meta);
}
}
@@ -92,32 +87,27 @@ export class EventClient extends PuterClient {
*
* Listeners run sequentially in the order they're registered so an earlier
* handler's mutation is visible to later ones. A listener that throws is
* logged (same as `emit`) and the chain continues.
* logged and the chain continues, with `meta.listener_failed` set.
*/
async emitAndWait<T extends keyof EventMap>(
key: T,
data: EventMap[T],
meta: EventMetadata,
) {
if (!this.hasListeners(key)) return;
// Spanned because callers block on listeners — this is where time
// spent in extension hooks (e.g. `ip.validate`) gets attributed.
return withSpan('event.emitAndWait', { 'event.key': key }, async () => {
const parts = key.split('.');
for (let i = 0; i < parts.length; i++) {
const matchKey = (
i === parts.length - 1
? key
: `${parts.slice(0, i + 1).join('.')}.*`
) as ListenKey;
const extensionListeners = extensionStore.events[matchKey];
const listeners = (this.#eventListeners[matchKey] || []).concat(
extensionListeners || [],
);
if (!listeners) continue;
for (const listener of listeners) {
for (const matchKey of this.#matchKeys(key)) {
// A wildcard listener observes the event; it does not implement it.
const isExact = matchKey === key;
for (const listener of this.#listenersFor(matchKey)) {
try {
await listener(key, data, meta);
} catch (e) {
// The chain continues, but the caller is told: a hook
// that could not run is not a hook that approved.
if (isExact) meta.listener_failed = true;
console.error(
'Error in event listener for event',
key,
@@ -129,6 +119,43 @@ export class EventClient extends PuterClient {
});
}
/**
* Like `emitAndWait`, but every matched listener runs at once, so none sees
* another's mutation of `data` before all have finished. Only for events
* whose listeners each add an independent share, written after their own
* awaits (`const v = await x; data.n += v`).
*/
async emitAndWaitConcurrently<T extends keyof EventMap>(
key: T,
data: EventMap[T],
meta: EventMetadata,
) {
const matched = this.#matchKeys(key).flatMap((matchKey) =>
this.#listenersFor(matchKey).map((listener) => ({
listener,
// A wildcard listener observes the event; it does not implement it.
isExact: matchKey === key,
})),
);
if (matched.length === 0) return;
return withSpan('event.emitAndWait', { 'event.key': key }, async () => {
await Promise.all(
matched.map(async ({ listener, isExact }) => {
try {
await listener(key, data, meta);
} catch (e) {
if (isExact) meta.listener_failed = true;
console.error(
'Error in event listener for event',
key,
e,
);
}
}),
);
});
}
/**
* Whether anything would run if `key` were emitted, wildcards included.
*
@@ -137,13 +164,7 @@ export class EventClient extends PuterClient {
* otherwise perfectly cheap and doesn't need this.
*/
hasListeners<T extends keyof EventMap>(key: T): boolean {
const parts = key.split('.');
for (let i = 0; i < parts.length; i++) {
const matchKey = (
i === parts.length - 1
? key
: `${parts.slice(0, i + 1).join('.')}.*`
) as ListenKey;
for (const matchKey of this.#matchKeys(key)) {
if (this.#eventListeners[matchKey]?.length) return true;
if (extensionStore.events[matchKey]?.length) return true;
}
@@ -186,6 +207,34 @@ export class EventClient extends PuterClient {
const idx = listeners.indexOf(callback as EventListener);
if (idx !== -1) listeners.splice(idx, 1);
}
/**
* The keys a listener can be registered under to hear `key`: each
* `<prefix>.*` shorter than the key, then the key itself. Memoized, since
* the same few keys are emitted on every request.
*/
#matchKeys(key: string): readonly ListenKey[] {
const cached = this.#matchKeyMemo.get(key);
if (cached) return cached;
const parts = key.split('.');
const matchKeys = parts.map(
(_, i) =>
(i === parts.length - 1
? key
: `${parts.slice(0, i + 1).join('.')}.*`) as ListenKey,
);
this.#matchKeyMemo.set(key, matchKeys);
return matchKeys;
}
#listenersFor(matchKey: ListenKey): readonly EventListener[] {
const own = this.#eventListeners[matchKey];
const fromExtensions = extensionStore.events[matchKey];
if (!own?.length && !fromExtensions?.length) return NO_LISTENERS;
// A copy, so a listener that unsubscribes mid-dispatch can't make the
// loop skip its neighbour.
return (own ?? []).concat(fromExtensions ?? []);
}
async #emitEvent<T extends keyof EventMap>(
listener: EventListener,
key: T,
+111 -9
View File
@@ -175,6 +175,7 @@ export type EventMap = {
index_url?: string | null;
owner_user_id?: number | null;
name?: string | null;
godmode?: boolean;
};
};
@@ -267,6 +268,17 @@ export type EventMap = {
message: string | null;
[key: string]: unknown;
};
// Inbound mail to a feedback address (`abuse@`, `fbl@`, ...), emitted via
// `emitAndWait`. A listener that takes it sets `handled` and it reaches no
// mailbox; otherwise it is routed like any other message.
'email.ingress.feedback': {
/** The envelope recipient, as the relay presented it. */
to: string;
from: string | null;
/** The whole message, exactly as received. */
raw: Buffer;
handled: boolean;
};
'user.save_account': {
user_id: number;
old_username?: string;
@@ -293,6 +305,9 @@ export type EventMap = {
device_fingerprint: string | null;
allowed: boolean;
reason: string | null;
// With `allowed` false, offer the card fallback in place of SMS. Only
// honored while the fallback is on; otherwise the send proceeds.
card_fallback_instead?: boolean;
[key: string]: unknown;
};
// Fire-and-forget signal that a code was actually sent — the abuse
@@ -303,6 +318,9 @@ export type EventMap = {
user_uid: string;
phone: string;
device_fingerprint: string | null;
// Browser-signals dispatch id the client sent with this request, or
// null when it sent none.
dispatch_id: string | null;
};
'user.phone-verified': {
user_id: number;
@@ -322,6 +340,9 @@ export type EventMap = {
// the abuse extension cap card-verification setups per device (across
// accounts) before any Stripe SetupIntent is created.
device_fingerprint: string | null;
// 'trial' when the card is being added for a free trial; null for the
// verification gates.
purpose?: string | null;
enabled: boolean | null;
// Set false by the extension to refuse this setup (e.g. the per-device
// setup-velocity cap); `reason` carries the opaque code. Stays true
@@ -376,6 +397,8 @@ export type EventMap = {
user_id: number;
user_uuid?: string;
stripe_customer_id?: string | null;
/** Apps the account owned, which cascaded away with it. */
appUids?: string[];
};
// ---- Team billing ---- the trigger, never the charge ----
@@ -438,7 +461,11 @@ export type EventMap = {
[key: string]: unknown;
};
};
'storage.quota.bonus': { userId: number; extra: number };
/**
* Bytes to add to a user's storage allowance. Listeners run concurrently:
* add to `extra` after your own awaits, never `extra += await x`.
*/
'storage.quota.bonus': { userId: number; userUuid?: string; extra: number };
// ---- Key-value ----
/**
@@ -463,13 +490,35 @@ export type EventMap = {
values?: unknown[];
/** Keys among `keys` private to the namespace's app. */
noShareKeys?: string[];
/** Events handler runs behind the write; see `Actor.handlerDepth`. */
handlerDepth?: number;
/**
* The user who made the change. Differs from `userId` when a share
* holder wrote into the owner's namespace.
*/
actingUserId?: number;
/** Made by a worker session or an events handler; see `viaWorkerOf`. */
viaWorker?: true;
};
/**
* A whole namespace was emptied. Namespace-level on purpose: `flush`'s own
* key enumeration is truncated for a large namespace, so the keys it saw
* are not the keys it removed.
*/
'kv.flushed': { namespace: string; userId: number };
'kv.flushed': {
namespace: string;
userId: number;
actingUserId?: number;
viaWorker?: true;
};
/**
* Share handles stopped being addresses, whoever retired them. `userId` is
* one user who could have subscribed through them, their owner or their
* holder. Listeners end what was made through them even where another grant
* still covers the region. Local, like `permission.revoked`: one region
* settling covers every other.
*/
'kv.share.retired': { handles: string[]; userId: number };
// ---- Notifications ----
/**
@@ -486,6 +535,8 @@ export type EventMap = {
appUid: string | null;
value: Record<string, unknown>;
createdAt: number;
/** Events handler runs behind whatever created it. */
handlerDepth?: number;
};
/**
* One notification, addressed at whatever sockets each region holds for the
@@ -589,6 +640,11 @@ export type EventMap = {
generation: number;
/** Whether the table changed; only then does a peer need to re-read it. */
durable: boolean;
/**
* Session subscriptions this bump ended, so siblings stop queueing for
* them.
*/
ended?: string[];
};
/**
* A user's sockets moved between regions, so every node must drop what it
@@ -603,6 +659,12 @@ export type EventMap = {
userId: number;
generation: number;
};
/**
* Someone's entry was just shared, so every process must drop its cached
* "this owner has nothing shared" answer. `outer.pubsub.*` for the same
* reason as the events generation bump: a per-process cache.
*/
'outer.pubsub.share.ownerShared': { ownerUserId: number };
'outer.fs.write-hash': { hash: string; uuid: string };
/**
* Cache keys the KV read cache must stop serving, because the entries
@@ -639,6 +701,10 @@ export type EventMap = {
requestHash?: string;
mime: string;
};
// Asked for a request whose host is outside every configured domain. A
// listener that recognises the host sets `result.subdomain` to the site
// it serves; an unclaimed host is answered with a 404.
'site.resolveHost': { host: string; result: { subdomain: string | null } };
// Asked before a hosted file is streamed. A listener that owns the site
// sets `result.allowed = false` to withhold the entry; the visitor then
// sees the same 404 as for a missing file.
@@ -660,7 +726,8 @@ export type EventMap = {
uid?: string;
thumbnail?: string | null;
};
'thumbnail.created': { url: string };
/** `uuid` is the entry the thumbnail belongs to. */
'thumbnail.created': { url: string; uuid: string };
'thumbnail.upload.prepare': {
items: { index: number; item_uid: string }[];
uploadUrl?: string;
@@ -675,8 +742,19 @@ export type EventMap = {
* Sessions were revoked for a user. Anything holding one of them open — a
* live socket, say — should drop it: `revoked_at` is otherwise only read on
* the next handshake.
*
* `outer.*` rather than `outer.pubsub.*` on purpose: the eviction runs
* through the socket.io Redis adapter a cluster shares, so one node
* disconnecting covers every node in it — fanning out to siblings would
* just have each of them repeat the same cluster-wide disconnect.
*/
'auth.sessions.revoked': { user_id: number; session_uids: string[] };
'outer.auth.sessions.revoked': {
user_id: number;
session_uids: string[];
};
/** One access token was revoked; only its own connections should drop. */
'auth.access-token.revoked': { token_uid: string };
/**
* A grant was withdrawn, so whatever was standing on it has to be settled
@@ -708,6 +786,7 @@ export type EventMap = {
email?: string;
app_name?: string;
};
/** False for an app actor: add only `subscribed` / `paid_storage` then. */
isUser: boolean;
};
'wisp.get-policy': {
@@ -742,6 +821,12 @@ export type EventMap = {
// `ai.cost.factor.<driverName>.<provider>:<model>`. Emitted once per model
// per batch; the last listener to set `factor` wins.
[K in `ai.cost.factor.${string}`]: AiCostFactorEvent;
} & {
// A plan gate is about to refuse a caller. Keyed per surface the way the
// lifecycles above are: `subscription.gate.route.<method>.<path>`,
// `subscription.gate.driver.<iface>.<method>`. Emitted via `emitAndWait`;
// a listener that sets `allow` waives the plan requirement for that call.
[K in `subscription.gate.${string}`]: SubscriptionGateEvent;
} & {
[K in `pubsub.login.${string}`]: { authtoken: string };
} & {
@@ -762,6 +847,18 @@ export type EventMap = {
'outer.pubsub.metering.credits-changed': { userUuid: string };
} & IExtensionEventMap;
/** Payload for `subscription.gate.<surface>` events. */
export type SubscriptionGateEvent = {
actor: Actor;
/** The key after `subscription.gate.`, e.g. `driver.puter-kvstore.get`. */
surface: string;
/** What the gate asked for: any paid plan (`true`) or these policy ids. */
requirement: boolean | readonly string[];
/** The request being gated, when there is one. */
req?: ExpressRequest;
allow: boolean;
};
/** Payload for `ai.cost.factor.<driver>.<model>` events. */
export type AiCostFactorEvent = {
/** Driver doing the pricing, e.g. `ai-chat`. */
@@ -852,10 +949,11 @@ export type EventKey = keyof EventMap & string;
// Generates a wildcard for every non-final dot-separated prefix of K.
export type WildcardPrefixes<K extends string> =
K extends `${infer Head}.${infer Tail}`
? | `${Head}.*`
| (Tail extends `${string}.${string}`
? `${Head}.${WildcardPrefixes<Tail>}`
: never)
?
| `${Head}.*`
| (Tail extends `${string}.${string}`
? `${Head}.${WildcardPrefixes<Tail>}`
: never)
: never;
export type ListenKey = EventKey | WildcardPrefixes<EventKey>;
@@ -864,7 +962,11 @@ export type MatchingEvents<P extends ListenKey> = P extends `${infer Prefix}.*`
? Extract<EventKey, `${Prefix}.${string}`>
: P & EventKey;
export type EventMetadata = { from_outside?: boolean };
export type EventMetadata = {
from_outside?: boolean;
/** Set by `emitAndWait` when a listener threw, so a verdict hook can refuse. */
listener_failed?: boolean;
};
export type EventListener<K extends EventKey = EventKey> = (
key: K,
data: EventMap[K],
+62 -10
View File
@@ -3,18 +3,19 @@
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
* Puter is free software: you can redistribute it and/or modify it under the
* terms of the GNU Affero General Public License as published by the Free
* Software Foundation, either version 3 of the License, or (at your option) any
* later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
* This program is distributed in the hope that it will be useful, but WITHOUT
* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
* FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
* details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
* along with this program. If not, see
* [https://www.gnu.org/licenses/](https://www.gnu.org/licenses/).
*/
import { GetObjectCommand } from '@aws-sdk/client-s3';
@@ -24,7 +25,7 @@ import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import type { IConfig } from '../../types';
import { S3Client } from './S3Client';
import { S3Client, withPresignHeadroom } from './S3Client';
const LOCAL_BUCKET = 'puter-local';
@@ -487,3 +488,54 @@ describe('S3Client — short reads during multipart', () => {
expect(parts).toEqual([10, 10]);
});
});
describe('withPresignHeadroom', () => {
const HOUR = 60 * 60 * 1000;
const creds = (msLeft?: number, id = 'a') => ({
accessKeyId: id,
secretAccessKey: 's',
...(msLeft === undefined
? {}
: { expiration: new Date(Date.now() + msLeft) }),
});
it('reports expiry an hour early so the SDK refreshes while an hour remains', async () => {
const chain = vi.fn(async () => creds(5 * HOUR));
const identity = await withPresignHeadroom(chain)();
expect(chain).toHaveBeenCalledTimes(1);
const left = identity.expiration!.getTime() - Date.now();
expect(left).toBeGreaterThan(4 * HOUR - 1000);
expect(left).toBeLessThanOrEqual(4 * HOUR);
});
it('force-refreshes credentials that are inside the presign window', async () => {
const chain = vi.fn(async (options?: { forceRefresh?: boolean }) =>
options?.forceRefresh
? creds(6 * HOUR, 'fresh')
: creds(30 * 60 * 1000, 'stale'),
);
const identity = await withPresignHeadroom(chain)();
expect(chain).toHaveBeenLastCalledWith(
expect.objectContaining({ forceRefresh: true }),
);
expect(identity.accessKeyId).toBe('fresh');
});
it('leaves long-lived credentials alone', async () => {
const chain = vi.fn(async () => creds(undefined));
const identity = await withPresignHeadroom(chain)();
expect(chain).toHaveBeenCalledTimes(1);
expect(identity.expiration).toBeUndefined();
});
it('never reports an expiry inside the SDK refresh margin', async () => {
const chain = vi.fn(async () => creds(20 * 60 * 1000));
const identity = await withPresignHeadroom(chain)();
const left = identity.expiration!.getTime() - Date.now();
expect(left).toBeGreaterThan(5 * 60 * 1000);
});
});
+40 -1
View File
@@ -43,6 +43,43 @@ const DEFAULT_MULTIPART_PART_SIZE_BYTES = 5 * 1024 * 1024;
const FAUXQS_SAFE_PUT_OBJECT_LIMIT_BYTES = 10 * 1024 * 1024;
const LEGACY_STORAGE_BUCKET = 'puter-local';
// A presigned URL stops working when the credentials that signed it expire.
// Presigns last up to an hour, but the SDK only refreshes temporary
// credentials five minutes before expiry.
const PRESIGN_MAX_LIFETIME_MS = 60 * 60 * 1000;
const SDK_REFRESH_MARGIN_MS = 5 * 60 * 1000;
type CredentialChain = ReturnType<typeof fromNodeProviderChain>;
/**
* Keeps temporary credentials at least an hour from expiry when they sign. The
* SDK is handed each identity with its expiration moved an hour early, so its
* own refresh comes back here while an hour remains, and a credential already
* inside that window is force-refreshed from the chain.
*/
export function withPresignHeadroom(chain: CredentialChain): CredentialChain {
return async (options) => {
let identity = await chain(options);
const remaining = (i: typeof identity) =>
i.expiration ? i.expiration.getTime() - Date.now() : Infinity;
if (
remaining(identity) <
PRESIGN_MAX_LIFETIME_MS + SDK_REFRESH_MARGIN_MS
) {
identity = await chain({ ...options, forceRefresh: true });
}
if (!identity.expiration) return identity;
// If even fresh credentials are short-lived, report an expiry just past
// the SDK's margin rather than one in the past, which would make it
// re-resolve on every request.
const reported = Math.max(
identity.expiration.getTime() - PRESIGN_MAX_LIFETIME_MS,
Date.now() + SDK_REFRESH_MARGIN_MS + 60 * 1000,
);
return { ...identity, expiration: new Date(reported) };
};
}
type S3CommandSender = Pick<AwsS3Client, 'send'>;
export class S3Client extends PuterClient {
@@ -83,7 +120,9 @@ export class S3Client extends PuterClient {
if (useCredentialChain) {
this.useProviderChain = true;
this.awsConfig = { credentials: fromNodeProviderChain() };
this.awsConfig = {
credentials: withPresignHeadroom(fromNodeProviderChain()),
};
this.partSize = 64 * 1024 * 1024;
this.maxSingleUploadSize = 128 * 1024 * 1024;
} else {
+159
View File
@@ -0,0 +1,159 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { existsSync, readFileSync } from 'node:fs';
import path from 'node:path';
import type { IConfig } from './types';
// Config resolution order:
// 1. `process.env.PUTER_CONFIG_PATH` — absolute path to a config file. Used
// by deployments whose outer bootstrap writes a merged config to a known
// location.
// 2. `<PACKAGE_ROOT>/config.json` — user's runtime override (gitignored),
// deep-merged over config.default.json so users can omit keys they
// don't care to override (e.g. gui_assets_root, database).
// 3. `<PACKAGE_ROOT>/config.default.json` — bundled OSS defaults.
//
// Post-flatten depth: compiled file is at `packages/puter/dist/src/backend/index.js`,
// so three `..`s land at `packages/puter/`.
const PACKAGE_ROOT = path.resolve(__dirname, '../../..');
// Root of the running code tree. Matches PACKAGE_ROOT for a source run, but
// points at `dist/` for a compiled run — so config-declared paths like
// `./extensions` resolve to `dist/extensions` at runtime without the config
// having to know about the build layout.
const RUNTIME_ROOT = path.resolve(__dirname, '../..');
const isPlainObject = (v: unknown): v is Record<string, unknown> =>
typeof v === 'object' && v !== null && !Array.isArray(v);
const deepMerge = <T extends Record<string, unknown>>(
base: T,
override: Record<string, unknown>,
): T => {
const out: Record<string, unknown> = { ...base };
for (const [k, v] of Object.entries(override)) {
out[k] =
isPlainObject(v) && isPlainObject(out[k])
? deepMerge(out[k] as Record<string, unknown>, v)
: v;
}
return out as T;
};
export const loadConfig = (): IConfig => {
const envPath = process.env.PUTER_CONFIG_PATH;
const runtimePath = path.join(PACKAGE_ROOT, 'config.json');
const defaultPath = path.join(PACKAGE_ROOT, 'config.default.json');
const defaults = existsSync(defaultPath)
? (JSON.parse(readFileSync(defaultPath, 'utf8')) as Record<
string,
unknown
>)
: {};
// Runtime override path: env wins, then config.json, else no override
// (we still return defaults so single-file installs work).
const overridePath =
envPath && existsSync(envPath)
? envPath
: existsSync(runtimePath)
? runtimePath
: null;
console.log(`[config] defaults from ${defaultPath}`);
if (overridePath) console.log(`[config] override from ${overridePath}`);
const override = overridePath
? (JSON.parse(readFileSync(overridePath, 'utf8')) as Record<
string,
unknown
>)
: {};
const config = deepMerge(defaults, override) as IConfig;
if (!config.version) {
const pkgPath = path.join(PACKAGE_ROOT, 'package.json');
if (existsSync(pkgPath)) {
try {
const pkg = JSON.parse(readFileSync(pkgPath, 'utf8')) as {
version?: string;
};
if (pkg.version) config.version = pkg.version;
} catch {
// fall through — /version returns 'unknown'
}
}
}
// Computed defaults. `origin` and `pub_port` are the externally-visible
// URL+port — what the browser sees. Separate from `port`, which is the
// bind port (can differ when behind a reverse proxy). Code paths that
// build self-referential URLs (GUI bootstrap, email links, OIDC callbacks)
// depend on `origin` having the right port baked in.
if (config.pub_port === undefined) config.pub_port = config.port;
const protocol = config.protocol ?? 'http';
const domain = config.domain ?? 'localhost';
const portSuffix =
config.pub_port === 80 || config.pub_port === 443
? ''
: `:${config.pub_port}`;
if (config.origin === undefined) {
config.origin = `${protocol}://${domain}${portSuffix}`;
}
// API lives on the `api.` subdomain on the same host+port as the main
// origin (see PuterRouter subdomain handling in server.ts). Compute it
// from pub_port/domain so a single-port override (e.g. port: 5101) flows
// through to the GUI bootstrap without users having to restate the URL.
if (config.api_base_url === undefined) {
config.api_base_url = `${protocol}://api.${domain}${portSuffix}`;
}
// Resolve path-valued config fields. Two different roots:
// - `extensions` uses RUNTIME_ROOT because extensions ship inside the
// build output (dist/extensions) and the loader's dynamic import()
// resolves relative paths against the *importing* module file.
// - GUI/puter-js/builtin-apps use PACKAGE_ROOT because those assets
// live only in the source tree (not copied into dist/) and are served
// via express.static at runtime.
const resolveRuntime = (p: string): string =>
path.isAbsolute(p) ? p : path.resolve(RUNTIME_ROOT, p);
const resolvePackage = (p: string): string =>
path.isAbsolute(p) ? p : path.resolve(PACKAGE_ROOT, p);
if (Array.isArray(config.extensions)) {
config.extensions = config.extensions.map(resolveRuntime);
}
if (typeof config.gui_assets_root === 'string') {
config.gui_assets_root = resolvePackage(config.gui_assets_root);
}
if (typeof config.puterjs_root === 'string') {
config.puterjs_root = resolvePackage(config.puterjs_root);
}
if (isPlainObject(config.builtin_apps)) {
for (const [k, v] of Object.entries(config.builtin_apps)) {
if (typeof v === 'string') {
(config.builtin_apps as Record<string, string>)[k] =
resolvePackage(v);
}
}
}
return config;
};
+106 -123
View File
@@ -17,25 +17,21 @@
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { isAccessTokenActor, isAppActor } from '../../core/actor.js';
import {
isAccessTokenActor,
isAccountContext,
isAppActor,
} from '../../core/actor.js';
import { HttpError } from '../../core/http/HttpError.js';
import { driversContainers } from '../../exports.js';
import {
APP_ICON_SIZES,
ICON_DATA_URL_MIME_ALLOWLIST,
isTrustedIconHost,
} from '../../util/appIcon.js';
import { resolvePrivateLaunchAccess } from '../../util/privateLaunchAccess.js';
import { toAppReadView } from '../../util/appView.js';
import { PuterController } from '../types.js';
import DEFAULT_APP_ICON from './default-app-icon.js';
/**
* REST endpoints for app management.
*
* Delegates to AppDriver for the actual CRUD + permission logic — these routes
* are just thin shape adapters that translate REST conventions into driver
* calls.
*/
/**
* Desktop boot reads the app list and individual app records repeatedly, so the
* ceiling is set well above normal boot traffic and exists to catch a runaway
@@ -69,11 +65,19 @@ const APP_ICON_LIMIT = {
key: 'ip',
};
export class AppController extends PuterController {
get appStore() {
return this.stores.app;
}
// Shared by `GET /apps/:name` (pipe-separated) and `POST /query/app`, which
// both fan out one lookup per name.
const APP_LOOKUP_MAX_ENTRIES = 200;
const APP_LOOKUP_MAX_SELECTOR_LEN = 200;
/** What `/apps/:name` reports for an app no launch gate applies to. */
const PUBLIC_APP_ACCESS = { hasAccess: true, checkedBy: 'core/public-app' };
/**
* REST endpoints for app management. CRUD goes through the apps driver; the
* read routes use AppService for visibility and views, as the driver does.
*/
export class AppController extends PuterController {
// In-flight background app-open writes. Tracked only so tests and
// shutdown can wait for them — the request path never does.
#pendingOpenWrites = new Set();
@@ -125,15 +129,6 @@ export class AppController extends PuterController {
await Promise.allSettled([...this.#pendingOpenWrites]);
}
get appDriver() {
// Drivers are wired into the shared driversContainers export by
// PuterServer at boot. Controllers get them lazily via this getter
// since they're instantiated before drivers in the boot order.
const d = driversContainers.apps;
if (!d) throw new Error('AppDriver not registered yet');
return d;
}
registerRoutes(
/** @type {import('../../core/http/PuterRouter').PuterRouter} */ router,
) {
@@ -147,7 +142,7 @@ export class AppController extends PuterController {
rateLimit: APP_READ_LIMIT,
},
async (req, res) => {
const apps = await this.appDriver.select({
const apps = await this.drivers.apps.select({
predicate: ['user-can-edit'],
});
res.json(apps);
@@ -179,7 +174,7 @@ export class AppController extends PuterController {
{ legacyCode: 'bad_request' },
);
}
const available = await this.appDriver.isNameAvailable(name);
const available = await this.drivers.apps.isNameAvailable(name);
res.json({ name, available });
},
);
@@ -190,10 +185,10 @@ export class AppController extends PuterController {
// before 400-ing for a missing body field.
//
// Authorization: only two callers are trusted to report opens —
// 1. a root user actor (plain session, no `.app` and no access
// 1. the account acting as itself (plain session or full-access
// token), e.g. the GUI launching apps on behalf of the user;
// 2. the app-under-user actor for the app being reported.
// Everything else — access tokens (regardless of issuer), asset
// Everything else — scoped and app-issued access tokens, asset
// tokens, app actors reporting for a *different* app — is denied,
// otherwise any authenticated party could inflate another app's
// open count.
@@ -218,14 +213,11 @@ export class AppController extends PuterController {
});
}
// Access tokens (and any other non-user/non-app identity,
// e.g. asset tokens) are not allowed to report opens —
// they're shared / scoped credentials and shouldn't drive
// analytics counters.
if (isAccessTokenActor(actor)) {
// Shared credentials shouldn't drive analytics counters.
if (isAccessTokenActor(actor) && !isAccountContext(actor)) {
throw new HttpError(
403,
'Access tokens cannot report app opens',
'This token cannot report app opens',
{ legacyCode: 'forbidden' },
);
}
@@ -238,7 +230,7 @@ export class AppController extends PuterController {
);
}
const app = await this.appStore.getByUid(app_uid);
const app = await this.stores.app.getByUid(app_uid);
if (!app)
throw new HttpError(404, 'App not found', {
legacyCode: 'not_found',
@@ -266,37 +258,40 @@ export class AppController extends PuterController {
async (req, res) => {
const raw = req.params.name;
const names = raw.split('|').filter(Boolean);
if (names.length > APP_LOOKUP_MAX_ENTRIES) {
throw new HttpError(
400,
`at most ${APP_LOOKUP_MAX_ENTRIES} app names per request`,
{ legacyCode: 'bad_request' },
);
}
const userUid = req.actor?.user?.uuid ?? null;
const results = await Promise.all(
names.map(async (name) => {
const app = await this.appStore.getByName(name);
if (!app) return null;
let shaped;
try {
shaped = await this.appDriver.read({
uid: app.uid,
});
} catch {
return null;
}
const privateAccess = await resolvePrivateLaunchAccess({
app: shaped,
eventClient: this.clients.event,
userUid,
source: 'appsRoute',
args: req.query ?? {},
const actor = req.actor;
const appsByName = await this.stores.app.getByNames(
names.filter(
(name) => name.length <= APP_LOOKUP_MAX_SELECTOR_LEN,
),
);
const readable = await this.services.app.filterReadable(
[...new Set(appsByName.values())],
actor,
);
const views = await this.services.app.views(readable, actor, {
source: 'appsRoute',
});
const shapedByUid = new Map(
readable.map((app, i) => {
const shaped = toAppReadView(app, views[i], {
viewer: actor.user,
});
return {
...shaped,
privateAccess:
shaped.privateAccess?.hasAccess === false
? shaped.privateAccess
: privateAccess,
};
shaped.privateAccess ??= PUBLIC_APP_ACCESS;
return [app.uid, shaped];
}),
);
const results = names.map(
(name) =>
shapedByUid.get(appsByName.get(name)?.uid) ?? null,
);
// Single-name requests return the app directly; batch returns an array
if (names.length === 1) {
@@ -316,8 +311,8 @@ export class AppController extends PuterController {
//
// Access rules: only apps the caller has a legitimate reason to
// see are returned — public (`approved_for_listing`), owned by
// the caller, or explicitly accessible via AppDriver.read (for
// protected apps with a granted permission). Everything else is
// the caller, or readable per `AppService.filterReadable` (for
// protected apps, a granted permission). Everything else is
// silently skipped so the endpoint can't be used to enumerate
// existence of private / unapproved apps by guessing names.
//
@@ -327,9 +322,6 @@ export class AppController extends PuterController {
// included for public/owned apps only, consistent with
// marketplace semantics.
const QUERY_APP_MAX_ENTRIES = 200;
const QUERY_APP_MAX_SELECTOR_LEN = 200;
router.post(
'/query/app',
{
@@ -339,71 +331,62 @@ export class AppController extends PuterController {
},
async (req, res) => {
const appList = Array.isArray(req.body) ? req.body : [];
if (appList.length > QUERY_APP_MAX_ENTRIES) {
if (appList.length > APP_LOOKUP_MAX_ENTRIES) {
throw new HttpError(
400,
`request body must contain at most ${QUERY_APP_MAX_ENTRIES} selectors`,
`request body must contain at most ${APP_LOOKUP_MAX_ENTRIES} selectors`,
{ legacyCode: 'bad_request' },
);
}
const actorUserId = req.actor?.user?.id ?? null;
const results = [];
const actor = req.actor;
const selectors = appList.filter(
(selector) =>
typeof selector === 'string' &&
selector.length > 0 &&
selector.length <= APP_LOOKUP_MAX_SELECTOR_LEN,
);
const isUid = (selector) => selector.startsWith('app-');
const [byUid, byName] = await Promise.all([
this.stores.app.getByUids(selectors.filter(isUid)),
this.stores.app.getByNames(
selectors.filter((s) => !isUid(s)),
),
]);
const found = selectors
.map((s) => (isUid(s) ? byUid.get(s) : byName.get(s)))
.filter(Boolean);
for (const selector of appList) {
if (
typeof selector !== 'string' ||
selector.length === 0 ||
selector.length > QUERY_APP_MAX_SELECTOR_LEN
) {
continue;
}
const isUid = selector.startsWith('app-');
const app = isUid
? await this.appStore.getByUid(selector)
: await this.appStore.getByName(selector);
if (!app) continue;
const isOwner =
actorUserId !== null &&
app.owner_user_id === actorUserId;
const isApproved = Boolean(app.approved_for_listing);
if (!isOwner && !isApproved) {
// Unapproved, non-owned — only surface if the
// caller has an explicit grant (purchased /
// permissioned). AppDriver.read enforces that
// via #canReadApp; a thrown 403 means "not
// accessible" and we treat it as "not found".
try {
const shaped = await this.appDriver.read({
uid: app.uid,
});
if (!shaped) continue;
} catch {
continue;
}
}
const assocRows = await this.clients.db.read(
'SELECT `type` FROM `app_filetype_association` WHERE `app_id` = ?',
[app.id],
// Unapproved apps the caller doesn't own surface only when
// the caller may read them (one batched grant check).
const isListed = (app) =>
app.owner_user_id === actor.user?.id ||
Boolean(app.approved_for_listing);
const readable = new Set(
await this.services.app.filterReadable(
[...new Set(found.filter((app) => !isListed(app)))],
actor,
),
);
const visible = found.filter(
(app) => isListed(app) || readable.has(app),
);
const filetypesByAppId =
await this.stores.app.getFiletypeAssociationsByIds(
visible.map((app) => app.id),
);
results.push({
uuid: app.uid,
name: app.name,
title: app.title,
description: app.description,
metadata: app.metadata,
tags:
typeof app.tags === 'string'
? app.tags.split(',')
: [],
created: app.timestamp,
associations: assocRows.map((r) => r.type),
});
}
const results = visible.map((app) => ({
uuid: app.uid,
name: app.name,
title: app.title,
description: app.description,
metadata: app.metadata,
tags:
typeof app.tags === 'string' ? app.tags.split(',') : [],
created: app.timestamp,
associations: filetypesByAppId.get(app.id) ?? [],
}));
res.json(results);
},
@@ -461,7 +444,7 @@ export class AppController extends PuterController {
}
if (!appUid.startsWith('app-')) appUid = `app-${appUid}`;
const app = await this.appStore.getByUid(appUid);
const app = await this.stores.app.getByUid(appUid);
const icon = app?.icon;
// If the icon isn't an inline data URL, try to serve it from the
@@ -338,6 +338,42 @@ describe('AppController POST /rao', () => {
expect(rows).toHaveLength(1);
});
it('keeps every open as a row, and counts one per account per hour', async () => {
const owner = await makeUser();
const other = await makeUser();
const app = await createApp(owner.actor);
const open = async (user: { actor: Actor; userId: number }) => {
const { res } = makeRes();
await withActor(user.actor, () =>
callRoute(
'post',
'/rao',
makeReq({ body: { app_uid: app.uid }, actor: user.actor }),
res,
),
);
};
for (let i = 0; i < 5; i++) await open(owner);
await open(other);
await server.controllers.apps.drainPendingAppOpens();
// Every open is still a row: the recent-apps list orders on them.
const rows = (await server.clients.db.read(
'SELECT `user_id` FROM `app_opens` WHERE `app_uid` = ?',
[app.uid],
)) as Array<{ user_id: number }>;
expect(rows).toHaveLength(6);
// The count is accounts-per-hour, so repeats do not move it.
const stats = await server.stores.app.getAppsStats([app.uid]);
expect(stats.get(app.uid)).toMatchObject({
open_count: 2,
user_count: 2,
});
});
// `app_opens` is analytics: the response carries nothing derived from it
// and the client never awaits the post. Holding the response open for a
// primary write put that write's latency inside the app launch it was
@@ -538,6 +574,57 @@ describe('AppController GET /apps/:name', () => {
});
});
it('throws 400 when the pipe-batch exceeds the 200-name cap', async () => {
const { actor } = await makeUser();
const { res } = makeRes();
const names = new Array(201).fill('x').join('|');
await expect(
withActor(actor, () =>
callRoute(
'get',
'/apps/:name',
makeReq({ params: { name: names }, actor }),
res,
),
),
).rejects.toMatchObject({ statusCode: 400 });
});
it('accepts a pipe-batch at exactly the 200-name cap', async () => {
const owner = await makeUser();
const app = await createApp(owner.actor);
const names = [app.name, ...new Array(199).fill('missing-app')].join(
'|',
);
const { res, captured } = makeRes();
await withActor(owner.actor, () =>
callRoute(
'get',
'/apps/:name',
makeReq({ params: { name: names }, actor: owner.actor }),
res,
),
);
expect(Array.isArray(captured.body)).toBe(true);
expect(captured.body as unknown[]).toHaveLength(200);
});
it('throws 404 for a single name over the per-name length cap', async () => {
const { actor } = await makeUser();
const { res } = makeRes();
await expect(
withActor(actor, () =>
callRoute(
'get',
'/apps/:name',
makeReq({ params: { name: 'x'.repeat(201) }, actor }),
res,
),
),
).rejects.toMatchObject({ statusCode: 404 });
});
it('omits the stale index_url for a non-owner', async () => {
const owner = await makeUser();
const stranger = await makeUser();
@@ -567,6 +654,48 @@ describe('AppController GET /apps/:name', () => {
expect(body.index_url).toBeUndefined();
expect(body.privateAccess).toMatchObject({ hasAccess: false });
});
it('asks the private-access hook once per private app and withholds its index_url', async () => {
const owner = await makeUser();
const stranger = await makeUser();
const app = await createApp(owner.actor, { name: uniqueName('priv') });
const other = await createApp(owner.actor, { name: uniqueName('pub') });
await server.clients.db.write(
'UPDATE `apps` SET `is_private` = 1 WHERE `uid` = ?',
[app.uid],
);
await server.stores.app.invalidateByUid(app.uid as string);
const emit = vi.spyOn(server.clients.event, 'emitAndWait');
try {
const { res, captured } = makeRes();
await withActor(stranger.actor, () =>
callRoute(
'get',
'/apps/:name',
makeReq({
params: { name: `${app.name}|${other.name}` },
actor: stranger.actor,
}),
res,
),
);
const [priv, pub] = captured.body as Array<Record<string, unknown>>;
expect(priv!.index_url).toBeUndefined();
expect(priv!.privateAccess).toMatchObject({ hasAccess: false });
expect(pub!.privateAccess).toEqual({
hasAccess: true,
checkedBy: 'core/public-app',
});
expect(
emit.mock.calls.filter(
(call) => call[0] === 'app.privateAccess.resolveLaunch',
),
).toHaveLength(1);
} finally {
emit.mockRestore();
}
});
});
// ── POST /query/app ─────────────────────────────────────────────────
@@ -741,6 +870,61 @@ describe('AppController POST /query/app', () => {
expect(body).toHaveLength(1);
expect(body[0]?.name).toBe(app.name);
});
it('resolves every selector with batched lookups, not one per selector', async () => {
const owner = await makeUser();
const apps = await Promise.all(
[0, 1, 2].map(() =>
createApp(owner.actor, {
name: uniqueName('batch'),
filetype_associations: ['.batchq'],
}),
),
);
const appStore = server.stores.app;
const spies = {
getByName: vi.spyOn(appStore, 'getByName'),
getByUid: vi.spyOn(appStore, 'getByUid'),
getFiletypeAssociations: vi.spyOn(
appStore,
'getFiletypeAssociations',
),
getByNames: vi.spyOn(appStore, 'getByNames'),
getByUids: vi.spyOn(appStore, 'getByUids'),
getFiletypeAssociationsByIds: vi.spyOn(
appStore,
'getFiletypeAssociationsByIds',
),
};
try {
const { res, captured } = makeRes();
await withActor(owner.actor, () =>
callRoute(
'post',
'/query/app',
makeReq({
body: [apps[0]!.name, apps[1]!.name, apps[2]!.uid],
actor: owner.actor,
}),
res,
),
);
const body = captured.body as Array<Record<string, unknown>>;
expect(body.map((item) => item.uuid)).toEqual(
apps.map((app) => app.uid),
);
expect(body[2]!.associations).toEqual(['batchq']);
expect(spies.getByName).not.toHaveBeenCalled();
expect(spies.getByUid).not.toHaveBeenCalled();
expect(spies.getFiletypeAssociations).not.toHaveBeenCalled();
expect(spies.getByNames).toHaveBeenCalledTimes(1);
expect(spies.getByUids).toHaveBeenCalledTimes(1);
expect(spies.getFiletypeAssociationsByIds).toHaveBeenCalledTimes(1);
} finally {
for (const spy of Object.values(spies)) spy.mockRestore();
}
});
});
// ── GET /app-icon/:app_uid(/:size) ──────────────────────────────────
@@ -897,6 +1081,68 @@ describe('AppController POST /rao actor gating', () => {
).rejects.toMatchObject({ statusCode: 403, legacyCode: 'forbidden' });
});
it('records an open for a full-access token — the account itself', async () => {
const owner = await makeUser();
const app = await createApp(owner.actor);
const patActor = makeActor({
user: owner.actor.user,
accessToken: {
uid: uuidv4(),
issuer: owner.actor,
authorized: null,
fullAccess: true,
},
});
const { res, captured } = makeRes();
await withActor(patActor, () =>
callRoute(
'post',
'/rao',
makeReq({ body: { app_uid: app.uid }, actor: patActor }),
res,
),
);
expect(captured.body).toEqual({});
await server.controllers.apps.drainPendingAppOpens();
const rows = (await server.clients.db.read(
'SELECT `app_uid` FROM `app_opens` WHERE `app_uid` = ? AND `user_id` = ?',
[app.uid, owner.userId],
)) as unknown[];
expect(rows).toHaveLength(1);
});
it('refuses a full-access claim on an app-issued token', async () => {
const { actor } = await makeUser();
const app = await createApp(actor);
const issuer = makeActor({ ...actor, app: { uid: app.uid as string } });
const tokenActor = makeActor({
user: actor.user,
accessToken: {
uid: uuidv4(),
issuer,
authorized: null,
fullAccess: true,
},
});
const { res } = makeRes();
await expect(
withActor(tokenActor, () =>
callRoute(
'post',
'/rao',
makeReq({
body: { app_uid: app.uid },
actor: tokenActor,
}),
res,
),
),
).rejects.toMatchObject({ statusCode: 403, legacyCode: 'forbidden' });
});
it("refuses to let an app actor record another app's open", async () => {
const { actor } = await makeUser();
const own = await createApp(actor);
@@ -17,11 +17,17 @@
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { afterAll, beforeAll, describe, expect, it } from 'vitest';
import { TOTP } from 'otpauth';
import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest';
import { makeActor } from '../../core/actor.js';
import { runWithContext } from '../../core/context.js';
import { createSecret as otpCreateSecret } from '../../services/auth/OTPUtil.js';
import { FULL_API_ACCESS } from '../../services/permission/consts.js';
import { setupPuterTestEnv, type PuterTestEnv } from '../../testUtil.js';
import {
createTestUser,
setupPuterTestEnv,
type PuterTestEnv,
} from '../../testUtil.js';
/**
* HTTP-level coverage for `/auth/revoke-own-access-token` — the route that
@@ -343,4 +349,369 @@ describe('revoke-own-access-token over HTTP', () => {
}
expect(await tokenReadStatus(file.uid, readToken)).not.toBe(200);
});
// These three run last in the file's shared `env.users.user`: each one
// revokes a session belonging to `owner.token` / `owner.workerToken`, so
// later tests can't rely on those credentials still being live.
it('a revoked access token yields 401 with no reauth_token or auth_id', async () => {
const owner = env.users.user;
const file = await makeFile(owner);
const readToken = await mintReadToken(owner.token, file.uid);
const revoke = await call(
'POST',
'/auth/revoke-own-access-token',
owner.token,
{ token: readToken },
);
expect(revoke.status).toBe(200);
const res = await call('GET', '/whoami', readToken);
expect(res.status).toBe(401);
const body = (await res.json()) as Record<string, unknown>;
expect(body.code).toBe('reauth_required');
expect(body.reauth_token).toBeUndefined();
expect(body.auth_id).toBeUndefined();
});
it('a revoked worker session gets no reauth_token even though it rides the session token type', async () => {
const owner = env.users.user;
const decoded = env.server.services.token.verify(
'auth',
owner.workerToken,
) as { session_uid: string };
await env.server.stores.session.removeByUuid(decoded.session_uid);
const res = await call('GET', '/whoami', owner.workerToken);
expect(res.status).toBe(401);
const body = (await res.json()) as Record<string, unknown>;
expect(body.code).toBe('reauth_required');
expect(body.reauth_token).toBeUndefined();
expect(body.auth_id).toBeUndefined();
});
it('a revoked GUI session still gets a reauth_token', async () => {
const owner = env.users.user;
const decoded = env.server.services.token.verify('auth', owner.token) as {
session_uid: string;
};
await env.server.stores.session.removeByUuid(decoded.session_uid);
const res = await call('GET', '/whoami', owner.token);
expect(res.status).toBe(401);
const body = (await res.json()) as Record<string, unknown>;
expect(body.code).toBe('reauth_required');
expect(typeof body.reauth_token).toBe('string');
const user = await env.server.stores.user.getByUsername(
owner.username,
);
expect(
env.server.services.auth.verifyReauthToken(
body.reauth_token as string,
).authId,
).toBe(user!.uuid);
});
});
/**
* Enrolling a second factor: setup sits behind the user-protected gate, and
* enable needs a live code for the secret setup issued.
*/
describe('2FA enrollment over HTTP', () => {
let env: PuterTestEnv;
beforeAll(async () => {
env = await setupPuterTestEnv();
}, 120_000);
afterAll(async () => {
await env?.shutdown();
});
const PASSWORD = 'tfa-http-password';
const uniq = () => Math.random().toString(36).slice(2, 10);
/** A signed-in password account with a confirmed address. */
const makeAccount = async () => {
const username = `tfa_${uniq()}`;
const { token } = await createTestUser(env.server, {
username,
password: PASSWORD,
});
const row = (await env.server.stores.user.getByUsername(username))!;
await env.server.stores.user.update(row.id, {
email: `${username}@test.local`,
email_confirmed: 1,
});
await env.server.stores.user.invalidateById(row.id);
return { username, token, userId: row.id };
};
/** The user-protected gate takes the session cookie on the GUI origin. */
const protectedPost = (
path: string,
token: string,
body: Record<string, unknown>,
credential: 'cookie' | 'bearer' = 'cookie',
) =>
fetch(new URL(path, env.origin), {
method: 'POST',
headers: {
'content-type': 'application/json',
...(credential === 'cookie'
? { cookie: `puter_auth_token=${token}` }
: { authorization: `Bearer ${token}` }),
},
body: JSON.stringify(body),
});
const setup = (
token: string,
body: Record<string, unknown>,
credential: 'cookie' | 'bearer' = 'cookie',
) => protectedPost('/user-protected/setup-2fa', token, body, credential);
const configure = (
token: string,
action: string,
body: Record<string, unknown> = {},
) =>
fetch(new URL(`/auth/configure-2fa/${action}`, env.apiOrigin), {
method: 'POST',
headers: {
'content-type': 'application/json',
authorization: `Bearer ${token}`,
},
body: JSON.stringify(body),
});
const otpState = async (userId: number) => {
const row = await env.server.stores.user.getByProperty('id', userId, {
force: true,
});
return {
secret: row?.otp_secret ?? null,
enabled: Boolean(row?.otp_enabled),
};
};
const liveTotp = (username: string, secret: string) =>
new TOTP({
issuer: 'puter.com',
label: username,
algorithm: 'SHA1',
digits: 6,
secret,
}).generate();
it('refuses setup to a session that has not re-proved the password', async () => {
const account = await makeAccount();
const noPassword = await setup(account.token, {});
expect(noPassword.status).toBe(403);
expect(await noPassword.text()).toContain('password_required');
const wrongPassword = await setup(account.token, { password: 'nope' });
expect(wrongPassword.status).toBe(400);
expect(await wrongPassword.text()).toContain('password_mismatch');
// Right password, but a bearer token is not the session cookie.
const bearer = await setup(
account.token,
{ password: PASSWORD },
'bearer',
);
expect(bearer.status).toBe(401);
// The old ungated path no longer hands out a secret.
expect((await configure(account.token, 'setup')).status).toBe(400);
expect((await otpState(account.userId)).secret).toBeNull();
});
it('enrolls after the password, and enables only on a live code', async () => {
const account = await makeAccount();
const res = await setup(account.token, { password: PASSWORD });
expect(res.status).toBe(200);
const { secret, codes } = (await res.json()) as {
secret: string;
codes: string[];
};
expect(codes).toHaveLength(10);
expect((await otpState(account.userId)).secret).toBe(secret);
expect((await configure(account.token, 'enable')).status).toBe(400);
const foreign = otpCreateSecret(account.username).secret;
const wrong = await configure(account.token, 'enable', {
code: liveTotp(account.username, foreign),
});
expect(wrong.status).toBe(400);
expect(await wrong.text()).toContain('code_mismatch');
expect((await otpState(account.userId)).enabled).toBe(false);
const enabled = await configure(account.token, 'enable', {
code: liveTotp(account.username, secret),
});
expect(enabled.status).toBe(200);
expect((await otpState(account.userId)).enabled).toBe(true);
});
});
/**
* A temp account re-attaches through `/signup` only from a session that ran
* out, not from one that was revoked.
*/
describe('temp-account reauth over HTTP', () => {
let env: PuterTestEnv;
beforeAll(async () => {
env = await setupPuterTestEnv();
}, 120_000);
afterAll(async () => {
await env?.shutdown();
}, 120_000);
const signup = (body: Record<string, unknown>) =>
fetch(new URL('/signup', env.origin), {
method: 'POST',
headers: { 'content-type': 'application/json' },
body: JSON.stringify(body),
});
it('a revoked temp session cannot be revived with its reauth_token', async () => {
const first = await signup({ is_temp: true });
expect(first.status).toBe(200);
const { token } = (await first.json()) as { token: string };
const decoded = env.server.services.token.verify('auth', token) as {
session_uid: string;
};
await env.server.services.auth.revokeSession(decoded.session_uid);
const who = await fetch(new URL('/whoami', env.apiOrigin), {
headers: { authorization: `Bearer ${token}` },
});
expect(who.status).toBe(401);
const { reauth_token } = (await who.json()) as {
reauth_token?: string;
};
expect(typeof reauth_token).toBe('string');
const revive = await signup({ is_temp: true, reauth_token });
expect(revive.status).toBe(401);
});
it('an expired temp session re-attaches to the same account', async () => {
const first = await signup({ is_temp: true });
expect(first.status).toBe(200);
const { user } = (await first.json()) as { user: { uuid: string } };
const reauth_token = env.server.services.auth.signReauthToken(
user.uuid,
'session_expired',
);
const again = await signup({ is_temp: true, reauth_token });
expect(again.status).toBe(200);
const body = (await again.json()) as {
token: string;
user: { uuid: string };
};
expect(body.user.uuid).toBe(user.uuid);
const who = await fetch(new URL('/whoami', env.apiOrigin), {
headers: { authorization: `Bearer ${body.token}` },
});
expect(who.status).toBe(200);
});
});
describe('session token helpers over HTTP', () => {
let env: PuterTestEnv;
beforeAll(async () => {
env = await setupPuterTestEnv();
}, 120_000);
afterAll(async () => {
await env?.shutdown();
}, 120_000);
const signupTemp = async () => {
const res = await fetch(new URL('/signup', env.origin), {
method: 'POST',
headers: { 'content-type': 'application/json' },
body: JSON.stringify({ is_temp: true }),
});
expect(res.status).toBe(200);
return (await res.json()) as { token: string; user: { uuid: string } };
};
const get = (path: string, token: string) =>
fetch(new URL(path, env.origin), {
headers: { authorization: `Bearer ${token}` },
});
const claims = (token: string) =>
env.server.services.token.verify('auth', token) as {
type: string;
user_uid: string;
session_uid: string;
};
it('mint tokens for the session the request authenticated with', async () => {
const { token, user } = await signupTemp();
const session = claims(token);
const row = await env.server.stores.user.getByUuid(user.uuid);
const getById = vi.spyOn(env.server.stores.user, 'getById');
try {
const gui = await get('/get-gui-token', token);
expect(gui.status).toBe(200);
const { token: guiToken } = (await gui.json()) as {
token: string;
};
expect(claims(guiToken)).toMatchObject({
type: 'gui',
user_uid: user.uuid,
session_uid: session.session_uid,
});
const sync = await get('/session/sync-cookie', token);
expect(sync.status).toBe(204);
const cookie = sync.headers.get('set-cookie') ?? '';
const cookieToken = /puter_auth_token=([^;]+)/.exec(cookie)?.[1];
expect(claims(cookieToken!)).toMatchObject({
type: 'session',
user_uid: user.uuid,
session_uid: session.session_uid,
});
// The user is the row token auth already read.
expect(getById.mock.calls.filter(([id]) => id === row!.id)).toEqual(
[],
);
} finally {
getById.mockRestore();
}
});
it("refuse a deleted account's session before either handler runs", async () => {
const { token, user } = await signupTemp();
const row = await env.server.stores.user.getByUuid(user.uuid);
await env.server.clients.db.write('DELETE FROM `user` WHERE `id` = ?', [
row!.id,
]);
await env.server.stores.user.markDeleted(row!);
expect((await get('/get-gui-token', token)).status).toBe(401);
const sync = await get('/session/sync-cookie', token);
expect(sync.status).toBe(401);
expect(sync.headers.get('set-cookie') ?? '').not.toContain(
'puter_auth_token=',
);
});
});
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
@@ -0,0 +1,371 @@
import { TOTP } from 'otpauth';
import { v4 as uuidv4 } from 'uuid';
import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest';
import {
createTestUser,
setupPuterTestEnv,
type PuterTestEnv,
} from '../../testUtil.js';
import { installSeatStub } from '../../testFixtures/seatStub.js';
describe('account change second-factor verification', () => {
let env: PuterTestEnv;
const password = 'email-change-password';
const links = new Map<string, string>();
beforeAll(async () => {
env = await setupPuterTestEnv();
vi.spyOn(env.server.clients.email, 'send').mockImplementation(
async (to, template, values) => {
if (template === 'email_verification_link')
links.set(to, String(values?.link));
return null;
},
);
}, 120_000);
afterAll(async () => {
await env?.shutdown();
});
const account = async (enabled = true, oidcOnly = false) => {
const username = `email2fa_${uuidv4().slice(0, 8)}`;
const credentials = await createTestUser(env.server, {
username,
password,
});
const user = (await env.server.stores.user.getByUsername(username))!;
const otp = new TOTP({
issuer: 'puter.com',
label: username,
algorithm: 'SHA1',
digits: 6,
});
await env.server.stores.user.update(user.id, {
email: `${username}@example.com`,
email_confirmed: true,
otp_enabled: enabled,
otp_secret: otp.secret.base32,
...(oidcOnly ? { password: null } : {}),
});
return { ...credentials, id: user.id, uuid: user.uuid, otp };
};
const change = (
token: string,
body: Record<string, unknown>,
revalidation?: string,
path = '/user-protected/change-email',
) =>
fetch(new URL(path, env.origin), {
method: 'POST',
headers: {
'content-type': 'application/json',
cookie: `puter_auth_token=${token}${revalidation ? `; puter_revalidation=${revalidation}` : ''}`,
},
body: JSON.stringify(body),
});
it.each([undefined, 'invalid-code', 123456, {}])(
'rejects an enabled account with code %j before staging a change',
async (code) => {
const user = await account();
const response = await change(user.token, {
password,
new_email: '[email protected]',
code,
});
expect(response.status).toBeGreaterThanOrEqual(400);
expect(response.status).toBeLessThan(500);
expect(await response.json()).toMatchObject({ factor: 'otp' });
const after = await env.server.stores.user.getById(user.id, {
force: true,
});
expect(after!.change_email_confirm_token).toBeFalsy();
expect(after!.unconfirmed_change_email).toBeFalsy();
},
);
it('rejects an incorrect six-digit code without staging the address', async () => {
const user = await account();
const code = ['000000', '000001', '000002', '000003'].find(
(token) => user.otp.validate({ token, window: 1 }) === null,
);
const response = await change(user.token, {
password,
new_email: '[email protected]',
code,
});
expect(response.status).toBe(401);
expect(await response.json()).toMatchObject({
code: 'code_mismatch',
factor: 'otp',
});
expect(
(await env.server.stores.user.getById(user.id, { force: true }))!
.change_email_confirm_token,
).toBeFalsy();
});
it('refuses a change when the enabled account secret is unavailable', async () => {
const user = await account();
await env.server.stores.user.update(user.id, {
otp_secret: 'invalid!',
});
expect(
(
await change(user.token, {
password,
new_email: '[email protected]',
code: '000000',
})
).status,
).toBe(401);
});
it('requires a code even after OIDC revalidation', async () => {
const user = await account(true, true);
const proof = env.server.services.token.sign(
'oidc-state',
{
user_uuid: user.uuid,
purpose: 'revalidate',
},
{ expiresIn: '5m' },
);
expect(
(
await change(
user.token,
{ new_email: '[email protected]' },
proof,
)
).status,
).toBe(400);
expect(
(
await change(
user.token,
{
new_email: '[email protected]',
code: user.otp.generate(),
},
proof,
)
).status,
).toBe(200);
});
it.each([true, false])(
'completes the change with the required factors (2FA enabled: %s)',
async (enabled) => {
const user = await account(enabled);
const newEmail = `${uuidv4()}@example.com`;
expect(
(
await change(user.token, {
password,
new_email: newEmail,
...(enabled ? { code: user.otp.generate() } : {}),
})
).status,
).toBe(200);
const link = links.get(newEmail)!;
expect(link).toBeTruthy();
expect((await fetch(link, { redirect: 'manual' })).status).toBe(
200,
);
expect(
(await env.server.stores.user.getById(user.id, {
force: true,
}))!.email,
).toBe(newEmail);
},
);
it('requires a new request if 2FA is enabled after the link was issued', async () => {
const user = await account(false);
const newEmail = `${uuidv4()}@example.com`;
expect(
(await change(user.token, { password, new_email: newEmail }))
.status,
).toBe(200);
await env.server.stores.user.update(user.id, { otp_enabled: true });
expect(
(await fetch(links.get(newEmail)!, { redirect: 'manual' })).status,
).toBe(400);
expect(
(await env.server.stores.user.getById(user.id, { force: true }))!
.email,
).not.toBe(newEmail);
});
it('refuses an existing confirmation link that never verified 2FA', async () => {
const user = await account();
const token = uuidv4();
await env.server.stores.user.update(user.id, {
change_email_confirm_token: token,
unconfirmed_change_email: '[email protected]',
});
const link = env.server.services.token.sign(
'otp',
{
token,
user_id: user.id,
purpose: 'change-email',
},
{ expiresIn: '1h' },
);
const response = await fetch(
new URL(`/change_email/confirm?token=${link}`, env.origin),
{ redirect: 'manual' },
);
expect(response.status).toBe(400);
expect(
(await env.server.stores.user.getById(user.id, { force: true }))!
.email,
).toBe(`${user.username}@example.com`);
});
describe('disabling 2FA', () => {
it.each([undefined, 'invalid-code', 123456, {}])(
'refuses code %j and retains the second-factor configuration',
async (code) => {
const user = await account();
await env.server.stores.user.update(user.id, {
otp_recovery_codes: 'one,two,three',
});
const response = await change(
user.token,
{ password, code },
undefined,
'/user-protected/disable-2fa',
);
expect(response.status).toBe(400);
expect(await response.json()).toMatchObject({ factor: 'otp' });
const after = await env.server.stores.user.getById(user.id, {
force: true,
});
expect(after!.otp_enabled).toBeTruthy();
expect(after!.otp_secret).toBe(user.otp.secret.base32);
expect(after!.otp_recovery_codes).toBe('one,two,three');
},
);
it('rejects an incorrect six-digit code', async () => {
const user = await account();
const code = ['000000', '000001', '000002', '000003'].find(
(token) => user.otp.validate({ token, window: 1 }) === null,
);
const response = await change(
user.token,
{ password, code },
undefined,
'/user-protected/disable-2fa',
);
expect(response.status).toBe(401);
expect(await response.json()).toMatchObject({
code: 'code_mismatch',
factor: 'otp',
});
expect(
(await env.server.stores.user.getById(user.id, {
force: true,
}))!.otp_enabled,
).toBeTruthy();
});
it.each([false, true])(
'accepts a current code after proving identity (OIDC-only: %s)',
async (oidcOnly) => {
const user = await account(true, oidcOnly);
const proof = oidcOnly
? env.server.services.token.sign(
'oidc-state',
{
user_uuid: user.uuid,
purpose: 'revalidate',
},
{ expiresIn: '5m' },
)
: undefined;
const missing = await change(
user.token,
oidcOnly ? {} : { password },
proof,
'/user-protected/disable-2fa',
);
expect(missing.status).toBe(400);
const response = await change(
user.token,
{
...(oidcOnly ? {} : { password }),
code: user.otp.generate(),
},
proof,
'/user-protected/disable-2fa',
);
expect(response.status).toBe(200);
const after = await env.server.stores.user.getById(user.id, {
force: true,
});
expect(after!.otp_enabled).toBeFalsy();
expect(after!.otp_secret).toBeNull();
expect(after!.otp_recovery_codes).toBeNull();
},
);
it('does not accept a code without password or OIDC revalidation', async () => {
const user = await account();
const response = await change(
user.token,
{ code: user.otp.generate() },
undefined,
'/user-protected/disable-2fa',
);
expect(response.status).toBe(403);
expect(await response.json()).toMatchObject({
code: 'password_required',
});
expect(
(await env.server.stores.user.getById(user.id, {
force: true,
}))!.otp_enabled,
).toBeTruthy();
});
it('does not let a valid code override a team requirement for 2FA', async () => {
const member = await account(true, true);
installSeatStub(env.server as never, [member.id], {
require2fa: true,
});
const proof = env.server.services.token.sign('oidc-state', {
user_uuid: member.uuid, purpose: 'revalidate',
}, { expiresIn: '5m' });
const response = await change(
member.token,
{ code: member.otp.generate() },
proof,
'/user-protected/disable-2fa',
);
expect(response.status).toBe(409);
expect(
(await env.server.stores.user.getById(member.id, {
force: true,
}))!.otp_enabled,
).toBeTruthy();
});
it('remains idempotent for an account with 2FA already disabled', async () => {
const user = await account(false);
expect(
(
await change(
user.token,
{ password },
undefined,
'/user-protected/disable-2fa',
)
).status,
).toBe(200);
});
});
});
@@ -0,0 +1,426 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { afterAll, beforeAll, describe, expect, it } from 'vitest';
import { isAccountContext } from '../../core/actor.js';
import type { AccessTokenPayload } from '../../services/auth/types.js';
import { setupPuterTestEnv, type PuterTestEnv } from '../../testUtil.js';
// Seeded with godmode on.
const DEV_CENTER = 'app-0b37f054-07d4-4627-8765-11bd23e889d4';
const TWELVE_HOURS = 12 * 60 * 60;
/**
* A godmode app launched from the desktop runs on a full-access token of its
* own: tied to the desktop session that asked for it, carrying the app for
* attribution, and expiring unless the desktop asks again.
*/
describe('godmode app tokens', () => {
let env: PuterTestEnv;
let userId = 0;
beforeAll(async () => {
env = await setupPuterTestEnv();
const row = await env.server.stores.user.getByUsername(
env.users.user.username,
);
userId = row!.id;
}, 120_000);
afterAll(async () => {
await env?.shutdown();
});
const call = (
method: string,
path: string,
token: string,
body?: unknown,
) =>
fetch(new URL(path, env.apiOrigin), {
method,
headers: {
'content-type': 'application/json',
authorization: `Bearer ${token}`,
},
...(body === undefined ? {} : { body: JSON.stringify(body) }),
});
const mint = async (sessionToken: string, appUid = DEV_CENTER) => {
const res = await call(
'POST',
'/auth/get-user-app-token',
sessionToken,
{ app_uid: appUid },
);
expect(res.status).toBe(200);
return (await res.json()) as {
token: string | null;
app_uid: string;
godmode?: boolean;
expires_at?: number;
};
};
const decode = (token: string) =>
env.server.services.token.verify<AccessTokenPayload>('auth', token);
const live = async (token: string) =>
(await env.server.services.auth.authenticate(token)).actor !== undefined;
/** A read token minted the way `getReadURL()` does, by `minter`. */
const mintScoped = async (minter: string) => {
const res = await call('POST', '/auth/create-access-token', minter, {
permissions: ['service:foo:ii:read'],
expiresIn: '7d',
});
expect(res.status).toBe(200);
return ((await res.json()) as { token: string }).token;
};
/** Rewrite a session row in place, as time passing would. */
const setRow = async (uuid: string, fields: Record<string, number>) => {
const cols = Object.keys(fields);
await env.server.clients.db.write(
`UPDATE \`sessions\` SET ${cols.map((c) => `\`${c}\` = ?`).join(', ')} WHERE \`uuid\` = ?`,
[...cols.map((c) => fields[c]), uuid],
);
await env.server.stores.session.publishCacheKeys({
keys: [`sessions:v2:uuid:${uuid}`],
});
};
/** A desktop session of its own, so revoking it leaves the shared one up. */
const newDesktopSession = async () => {
const user = await env.server.stores.user.getById(userId);
return env.server.services.auth.createSessionToken(user!);
};
/** An app row of its own, so demoting it disturbs nothing else. */
const makeApp = async ({
name = `gm-${crypto.randomUUID().slice(0, 8)}`,
godmode = true,
} = {}) => {
const uid = `app-${crypto.randomUUID()}`;
await env.server.clients.db.write(
'INSERT INTO `apps` (`uid`, `owner_user_id`, `name`, `title`, `index_url`, `godmode`, `timestamp`) VALUES (?, ?, ?, ?, ?, ?, CURRENT_TIMESTAMP)',
[uid, userId, name, 'Test App', `https://${name}.test/`, godmode ? 1 : 0],
);
return { uid, name };
};
it('issues a full-access token that carries the app', async () => {
const before = Math.floor(Date.now() / 1000);
const body = await mint(env.users.user.token);
expect(body).toMatchObject({ app_uid: DEV_CENTER, godmode: true });
expect(body.expires_at).toBeGreaterThanOrEqual(before + TWELVE_HOURS);
const claims = decode(body.token!);
expect(claims).toMatchObject({
type: 'access-token',
full_access: true,
godmode_app_uid: DEV_CENTER,
});
expect(claims.app_uid).toBeUndefined();
// The row's expiry is what lapses; the JWT itself never does.
expect(claims.exp).toBeUndefined();
const { actor } = await env.server.services.auth.authenticate(
body.token!,
);
expect(actor).toBeDefined();
expect(isAccountContext(actor!)).toBe(true);
expect(actor!.effectiveApp).toBeNull();
expect(actor!.accessToken).toMatchObject({
fullAccess: true,
godmodeApp: { uid: DEV_CENTER },
});
});
it('reaches the account surface but not account management', async () => {
const { token } = await mint(env.users.user.token);
expect((await call('GET', '/whoami', token!)).status).toBe(200);
expect((await call('GET', '/get-dev-profile', token!)).status).toBe(
200,
);
expect((await call('GET', '/auth/list-sessions', token!)).status).toBe(
403,
);
// It can't mint its own replacement; only the desktop can.
const self = await call('POST', '/auth/get-user-app-token', token!, {
app_uid: DEV_CENTER,
});
expect(self.status).toBe(403);
});
it('keeps one token per desktop session across renewals', async () => {
const desktop = await newDesktopSession();
const first = await mint(desktop.gui_token);
const second = await mint(desktop.gui_token);
expect(decode(second.token!).token_uid).toBe(
decode(first.token!).token_uid,
);
// Asked again within the hour: the row keeps the expiry it has.
expect(second.expires_at).toBe(first.expires_at);
// A copy taken at launch keeps working after the desktop renews.
expect(await live(first.token!)).toBe(true);
const row = await env.server.stores.session.getByUuidAny(
decode(second.token!).session_uid!,
);
expect(row).toMatchObject({
kind: 'access_token',
app_uid: DEV_CENTER,
expires_at: second.expires_at,
});
});
it('is extended once the desktop asks with under eleven hours left', async () => {
const desktop = await newDesktopSession();
const first = await mint(desktop.gui_token);
const sessionUid = decode(first.token!).session_uid!;
await setRow(sessionUid, {
expires_at: Math.floor(Date.now() / 1000) + 2 * 60 * 60,
});
const before = Math.floor(Date.now() / 1000);
const renewed = await mint(desktop.gui_token);
expect(decode(renewed.token!).token_uid).toBe(
decode(first.token!).token_uid,
);
expect(renewed.expires_at).toBeGreaterThanOrEqual(
before + TWELVE_HOURS,
);
const row = await env.server.stores.session.getByUuidAny(sessionUid);
expect(row?.expires_at).toBe(renewed.expires_at);
});
it('is never extended by being used', async () => {
const desktop = await newDesktopSession();
const { token, expires_at } = await mint(desktop.gui_token);
expect((await call('GET', '/whoami', token!)).status).toBe(200);
const row = await env.server.stores.session.getByUuidAny(
decode(token!).session_uid!,
);
expect(row?.expires_at).toBe(expires_at);
});
it('lists the token under its app for the account', async () => {
await mint(env.users.user.token);
const res = await call(
'GET',
'/auth/list-sessions',
env.users.user.token,
);
const sessions = (await res.json()) as Array<{
kind: string;
app_uid: string | null;
app: { name?: string } | null;
}>;
const row = sessions.find(
(s) => s.kind === 'access_token' && s.app_uid === DEV_CENTER,
);
expect(row?.app?.name).toBe('dev-center');
});
it('installs the app like any other launch, and uninstalls the same way', async () => {
const { uid } = await makeApp();
const app = await env.server.stores.app.getByUid(uid);
const installed = () =>
env.server.stores.permission.hasUserAppPerm(
userId,
app!.id,
'flag:app-is-authenticated',
);
const { token } = await mint(env.users.user.token, uid);
expect(await installed()).toBe(true);
expect(
await env.server.stores.fsEntry.getEntryByPath(
`/${env.users.user.username}/AppData/${uid}`,
),
).toBeTruthy();
// What the desktop's uninstall does: drop the grants, then the app's sessions.
const revoked = await call(
'POST',
'/auth/revoke-user-app',
env.users.user.token,
{ app_uid: uid, permission: '*' },
);
expect(revoked.status).toBe(200);
expect(await installed()).toBe(false);
await env.server.services.auth.revokeSession(decode(token!).session_uid!);
expect(await live(token!)).toBe(false);
});
it('dies with the desktop session that minted it', async () => {
const desktop = await newDesktopSession();
const { token } = await mint(desktop.gui_token);
expect((await call('GET', '/whoami', token!)).status).toBe(200);
await env.server.services.auth.revokeSession(
desktop.session.uuid as string,
);
const res = await call('GET', '/whoami', token!);
expect(res.status).toBe(401);
});
it('stops working once its app is no longer godmode', async () => {
const app = await makeApp();
const { token } = await mint(env.users.user.token, app.uid);
expect((await call('GET', '/whoami', token!)).status).toBe(200);
const row = await env.server.stores.app.getByUid(app.uid);
await env.server.clients.db.write(
'UPDATE `apps` SET `godmode` = 0 WHERE `uid` = ?',
[app.uid],
);
await env.server.stores.app.invalidate(row);
expect((await call('GET', '/whoami', token!)).status).toBe(401);
});
it('still gives an ordinary app an ordinary app token', async () => {
const app = await makeApp({ godmode: false });
const body = await mint(env.users.user.token, app.uid);
expect(body.godmode).toBeUndefined();
expect(decode(body.token!).type).toBe('app-under-user');
});
it('mints scoped tokens that are revoked along with it', async () => {
const { token } = await mint(env.users.user.token);
const child = await call(
'POST',
'/auth/create-access-token',
token!,
{ permissions: ['service:foo:ii:read'], expiresIn: '1h' },
);
expect(child.status).toBe(200);
const childToken = ((await child.json()) as { token: string }).token;
expect(
(await env.server.services.auth.authenticate(childToken)).actor,
).toBeDefined();
const full = await call('POST', '/auth/create-access-token', token!, {
permissions: ['full-api-access'],
});
expect(full.status).toBe(403);
const { actor } = await env.server.services.auth.authenticate(
env.users.user.token,
);
await env.server.services.auth.revokeAccessToken(actor!, token!);
expect(
(await env.server.services.auth.authenticate(childToken)).actor,
).toBeUndefined();
});
it('stays dead once it lapses; the desktop asking again gets a new one', async () => {
const desktop = await newDesktopSession();
const { token } = await mint(desktop.gui_token);
const child = await mintScoped(token!);
expect(await live(child)).toBe(true);
// The desktop stopped renewing it: its row's expiry passes.
await setRow(decode(token!).session_uid!, {
expires_at: Math.floor(Date.now() / 1000) - 1,
});
expect(await live(token!)).toBe(false);
expect(await live(child)).toBe(false);
const next = await mint(desktop.gui_token);
expect(decode(next.token!).token_uid).not.toBe(decode(token!).token_uid);
expect(await live(next.token!)).toBe(true);
expect(await live(token!)).toBe(false);
expect(await live(child)).toBe(false);
});
it('is replaced and revoked once past its max age, however often renewed', async () => {
const desktop = await newDesktopSession();
const { token } = await mint(desktop.gui_token);
const child = await mintScoped(token!);
await setRow(decode(token!).session_uid!, {
created_at: Math.floor(Date.now() / 1000) - 8 * 24 * 60 * 60,
});
const next = await mint(desktop.gui_token);
expect(decode(next.token!).token_uid).not.toBe(decode(token!).token_uid);
expect(await live(next.token!)).toBe(true);
expect(await live(token!)).toBe(false);
expect(await live(child)).toBe(false);
});
it('lets the tokens it minted die with the desktop session', async () => {
const desktop = await newDesktopSession();
const { token } = await mint(desktop.gui_token);
const child = await mintScoped(token!);
expect(await live(child)).toBe(true);
await env.server.services.auth.revokeSession(
desktop.session.uuid as string,
);
expect(await live(token!)).toBe(false);
expect(await live(child)).toBe(false);
});
it('reaches the AI wire routes, where a gate listener may waive the plan', async () => {
const { token } = await mint(env.users.user.token);
const route = '/puterai/openai/v1/chat/completions';
const send = (bearer: string) =>
fetch(new URL(route, env.apiOrigin), {
method: 'POST',
headers: {
'Content-Type': 'application/json',
Authorization: `Bearer ${bearer}`,
},
body: JSON.stringify({
model: 'fake',
provider: 'fake-chat',
messages: [{ role: 'user', content: 'hi' }],
max_tokens: 16,
}),
});
// Past the credential gates, stopped only by the plan.
expect((await send(token!)).status).toBe(402);
env.server.clients.event.on(
'subscription.gate.route.post.puterai.openai.v1.chat.completions',
(_key, event) => {
const gate = event as {
actor: { accessToken?: { godmodeApp?: { uid: string } } };
allow: boolean;
};
if (gate.actor.accessToken?.godmodeApp?.uid === DEV_CENTER) {
gate.allow = true;
}
},
);
expect((await send(token!)).status).not.toBe(402);
expect((await send(env.users.user.apiToken)).status).toBe(402);
});
});
@@ -0,0 +1,96 @@
import crypto from 'node:crypto';
import type { Response } from 'express';
import { HttpError } from '../../core/http/HttpError.js';
import type { OIDCService } from '../../services/auth/OIDCService.js';
import type { UserRow } from '../../stores/user/UserStore.js';
import type { IConfig } from '../../types.js';
import { sessionCookieFlags } from '../../util/cookieFlags.js';
export const OIDC_POPUP_RETURN_COOKIE_NAME = 'puter_oidc_popup_return';
export const OIDC_POPUP_RETURN_EXPIRY_SEC = 300;
export interface OidcLoginContinuation {
userUuid: string;
redirectUri: string;
embeddedInPopup: boolean;
openerOrigin: string | null;
messageId: string | null;
switchedToLogin: boolean;
}
interface OidcReturnDeps {
config: IConfig;
oidc: OIDCService;
}
/** Bind an attested popup return to the browser that completed sign-in. */
export function mintOidcPopupReturn(
res: Response,
payload: Record<string, unknown>,
deps: OidcReturnDeps,
): string {
const nonce = crypto.randomBytes(32).toString('base64url');
res.cookie(OIDC_POPUP_RETURN_COOKIE_NAME, nonce, {
...sessionCookieFlags(deps.config, { crossSite: false }),
httpOnly: true,
maxAge: OIDC_POPUP_RETURN_EXPIRY_SEC * 1000,
path: '/',
});
return deps.oidc.signPopupReturn({ action: 'sign-in', ...payload, nonce });
}
/** Call only after both the provider and the account's second factor succeed. */
export function completeOidcLoginReturn(
res: Response,
user: Pick<UserRow, 'uuid'>,
continuation: unknown,
deps: OidcReturnDeps,
): string {
if (!continuation || typeof continuation !== 'object') {
throw new HttpError(400, 'Invalid login continuation.', {
legacyCode: 'bad_request',
});
}
const data = continuation as OidcLoginContinuation;
if (
data.userUuid !== user.uuid ||
typeof data.redirectUri !== 'string' ||
typeof data.embeddedInPopup !== 'boolean' ||
typeof data.switchedToLogin !== 'boolean' ||
(data.openerOrigin !== null && typeof data.openerOrigin !== 'string') ||
(data.messageId !== null && typeof data.messageId !== 'string')
) {
throw new HttpError(400, 'Invalid login continuation.', {
legacyCode: 'bad_request',
});
}
const origin = (deps.config.origin ?? '').replace(/\/$/, '');
let target = data.redirectUri || origin || '/';
try {
if (origin && new URL(target).origin !== new URL(origin).origin)
target = origin;
} catch {
target = origin || '/';
}
const append = (key: string, value: string) => {
target += `${target.includes('?') ? '&' : '?'}${encodeURIComponent(key)}=${encodeURIComponent(value)}`;
};
if (data.embeddedInPopup) {
append('oidc_login', 'true');
append(
'opener_state',
mintOidcPopupReturn(
res,
{
opener_origin: data.openerOrigin,
msg_id: data.messageId,
oidc_login: true,
user_uuid: user.uuid,
},
deps,
),
);
}
if (data.switchedToLogin) append('oidc_switched', 'login');
return target;
}
@@ -0,0 +1,139 @@
import { v4 as uuidv4 } from 'uuid';
import {
afterAll,
afterEach,
beforeAll,
describe,
expect,
it,
vi,
} from 'vitest';
import { hashRecoveryCode } from '../../services/auth/OTPUtil.js';
import {
createTestUser,
setupPuterTestEnv,
type PuterTestEnv,
} from '../../testUtil.js';
describe('recovery code consumption', () => {
let env: PuterTestEnv;
const password = 'recovery-code-password';
beforeAll(async () => {
env = await setupPuterTestEnv();
}, 120_000);
afterAll(async () => {
await env?.shutdown();
});
afterEach(() => {
vi.restoreAllMocks();
});
// Hold the same cached snapshot until every concurrent request has read it.
const overlapReads = (userUuid: string, count: number) => {
const redis = env.server.clients.redis;
const get = redis.get.bind(redis);
const ready = Promise.withResolvers<void>();
let reads = 0;
vi.spyOn(redis, 'get').mockImplementation(async (key: string) => {
const value = await get(key);
if (key.includes(userUuid) && reads < count) {
if (++reads === count) ready.resolve();
await ready.promise;
}
return value;
});
};
const post = async (path: string, body: Record<string, unknown>) => {
const response = await fetch(new URL(path, env.origin), {
method: 'POST',
headers: { 'content-type': 'application/json' },
body: JSON.stringify(body),
});
return { response, body: await response.json() };
};
const account = async (codes: string[]) => {
const username = `recovery_${uuidv4().slice(0, 8)}`;
await createTestUser(env.server, { username, password });
const user = (await env.server.stores.user.getByUsername(username))!;
await env.server.stores.user.update(user.id, {
otp_enabled: true,
otp_secret: 'TESTSECRETBASE32',
otp_recovery_codes: codes.map(hashRecoveryCode).join(','),
});
const challenge = async () => {
const { response, body } = await post('/login', {
username,
password,
});
expect(response.status).toBe(202);
expect(body.next_step).toBe('otp');
return body.otp_jwt_token as string;
};
return { user, challenge };
};
it('allows only one concurrent redemption of the same code', async () => {
const { user, challenge } = await account(['RECOVER1', 'RECOVER2']);
const tokens = await Promise.all([challenge(), challenge()]);
overlapReads(user.uuid, tokens.length);
const results = await Promise.all(
tokens.map((token) =>
post('/login/recovery-code', { token, code: 'RECOVER1' }),
),
);
expect(
results.filter((result) => result.body.next_step === 'complete'),
).toHaveLength(1);
expect(
results.filter((result) => result.body.proceed === false),
).toHaveLength(1);
const after = await env.server.stores.user.getById(user.id, {
force: true,
});
expect(after!.otp_recovery_codes).toBe(hashRecoveryCode('RECOVER2'));
});
it('consumes different concurrent codes without restoring either one', async () => {
const codes = ['RECOVER1', 'RECOVER2', 'RECOVER3'];
const { user, challenge } = await account(codes);
const tokens = await Promise.all(codes.map(() => challenge()));
overlapReads(user.uuid, tokens.length);
const results = await Promise.all(
codes.map((code, index) =>
post('/login/recovery-code', { token: tokens[index], code }),
),
);
expect(
results.every((result) => result.body.next_step === 'complete'),
).toBe(true);
const after = await env.server.stores.user.getById(user.id, {
force: true,
});
expect(after!.otp_recovery_codes).toBe('');
const replay = await post('/login/recovery-code', {
token: tokens[0],
code: codes[0],
});
expect(replay.body).toMatchObject({ proceed: false });
});
it.each([{ code: 123456 }, { code: {} }, { code: [] }])(
'rejects a non-string recovery code $code without consuming a code',
async ({ code }) => {
const { user, challenge } = await account(['RECOVER1']);
const result = await post('/login/recovery-code', {
token: await challenge(),
code,
});
expect(result.response.status).toBe(400);
expect(
(await env.server.stores.user.getById(user.id, {
force: true,
}))!.otp_recovery_codes,
).toBe(hashRecoveryCode('RECOVER1'));
},
);
});
@@ -0,0 +1,82 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import type { Request, Response } from 'express';
import { v4 as uuidv4 } from 'uuid';
import { afterAll, beforeAll, describe, expect, it } from 'vitest';
import { configContainer } from '../../exports.js';
import { PuterServer } from '../../server.js';
import { setupTestServer } from '../../testUtil.js';
import type { IConfig } from '../../types.js';
import { startWebSession } from './webSession.js';
let server: PuterServer;
beforeAll(async () => {
server = await setupTestServer();
});
afterAll(async () => {
await server?.shutdown();
});
describe('startWebSession', () => {
it('records the request on the session, sets the HTTP-only cookie and returns a GUI token', async () => {
const username = `ws_${Math.random().toString(36).slice(2, 10)}`;
const user = await server.stores.user.create({
username,
uuid: uuidv4(),
password: null,
email: `${username}@test.local`,
});
const cookies: Array<{ name: string; value: string; opts: unknown }> =
[];
const res = {
cookie: (name: string, value: string, opts: unknown) => {
cookies.push({ name, value, opts });
},
} as unknown as Response;
const req = {
ip: '192.0.2.10',
headers: { 'user-agent': 'ws-agent', origin: 'https://gui.test' },
} as unknown as Request;
const config = configContainer as IConfig;
const guiToken = await startWebSession(req, res, user, {
config,
auth: server.services.auth,
});
expect(typeof guiToken).toBe('string');
expect(cookies).toHaveLength(1);
expect(cookies[0]!.name).toBe(config.cookie_name ?? 'puter_token');
expect(cookies[0]!.value).not.toBe(guiToken);
expect(cookies[0]!.opts).toMatchObject({ httpOnly: true });
const sessions = (await server.stores.session.getByUserId(
user.id,
)) as Array<Record<string, unknown>>;
expect(sessions).toHaveLength(1);
expect(sessions[0]).toMatchObject({
kind: 'web',
last_ip: '192.0.2.10',
last_user_agent: 'ws-agent',
});
});
});
@@ -0,0 +1,48 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import type { Request, Response } from 'express';
import type { AuthService } from '../../services/auth/AuthService.js';
import type { UserRow } from '../../stores/user/UserStore.js';
import type { IConfig } from '../../types.js';
import { sessionCookieFlags } from '../../util/cookieFlags.js';
/**
* Open a web session for a sign-in this request completed, recording where it
* came from, and set its HTTP-only cookie. Returns the GUI token, the only
* credential the client sees.
*/
export async function startWebSession(
req: Request,
res: Response,
user: UserRow,
deps: { config: IConfig; auth: AuthService },
): Promise<string> {
const { token, gui_token } = await deps.auth.createSessionToken(user, {
ip: req.ip || req.socket?.remoteAddress,
user_agent: req.headers?.['user-agent'],
origin: req.headers?.origin,
host: req.headers?.host,
});
res.cookie(deps.config.cookie_name ?? 'puter_token', token, {
...sessionCookieFlags(deps.config),
httpOnly: true,
});
return gui_token;
}
@@ -73,8 +73,8 @@ export class BroadcastController extends PuterController {
/**
* Event deliveries one region addressed at this one, on the same signed
* channel as the webhook above but carrying socket traffic rather than bus
* events. The answer names the pairs this region holds no socket for, which
* is what lets the sender correct its presence row.
* events. The answer names the anchor tokens this region holds no session
* row for, which is what lets the sender prune its remote-watch index.
*/
@Post('/events', { subdomain: '*' })
async events(req: Request, res: Response): Promise<void> {
@@ -46,17 +46,6 @@ const PREFERENCE_WRITE_LIMIT = {
};
export class DesktopController extends PuterController {
constructor(config, clients, stores, services) {
super(config, clients, stores, services);
}
get userStore() {
return this.stores.user;
}
get db() {
return this.clients.db;
}
registerRoutes(router) {
// -- Desktop background --------------------------------------
@@ -109,7 +98,7 @@ export class DesktopController extends PuterController {
});
}
await this.userStore.update(req.actor.user.id, patch);
await this.stores.user.update(req.actor.user.id, patch);
res.json({});
},
);
@@ -134,7 +123,7 @@ export class DesktopController extends PuterController {
);
}
await this.userStore.update(req.actor.user.id, {
await this.stores.user.update(req.actor.user.id, {
taskbar_items: JSON.stringify(items),
});
res.json({});
@@ -250,7 +239,7 @@ export class DesktopController extends PuterController {
const setClause = keys.map((k) => `\`${k}\` = ?`).join(', ');
const values = keys.map((k) => patch[k]);
await this.db.write(
await this.clients.db.write(
`UPDATE \`fsentries\` SET ${setClause} WHERE \`id\` = ?`,
[...values, entry.id],
);
@@ -262,6 +262,22 @@ describe('DriverController — concurrent acquire/release', () => {
expect(res2.body).toMatchObject({ success: true, result: 'pong' });
});
it('releases a slot acquired after the client has disconnected', async () => {
const handler = captureCallHandler(buildController(makeSyntheticDriver()));
const res1 = new StubRes();
Object.assign(res1, { destroyed: true });
await callInScope(handler,
makeReq({ interface: 'test-iface', method: 'ping' }, 'already-aborted'),
res1 as unknown as Response);
await new Promise((resolve) => setImmediate(resolve));
const res2 = new StubRes();
await callInScope(handler,
makeReq({ interface: 'test-iface', method: 'ping' }, 'already-aborted'),
res2 as unknown as Response);
res2.emit('finish');
expect(res2.body).toMatchObject({ success: true, result: 'pong' });
});
it('does not attach release listeners when the driver declares no concurrent config', async () => {
// With no spec there is nothing to release, so the gate must not
// wire `finish`/`close` release listeners. The client-disconnect
Loaded 100 of 1374 files, more files were not shown because too many files have changed in this diff. Show more