Compare commits

..
Author SHA1 Message Date
Daniel Salazar fd8d556cd5 Merge branch 'main' into refactor/ai-driver-layout 2026-11-13 23:05:37 -07:00
Daniel Salazar af01a554fe feat(gui): a top-up asked for from a page names the app it runs as (#4637)
The framed top-up passes the deployment's dialog the app the framing page's
origin resolves to, as the framed upgrade does. The dashboard link puter.js
opens when no frame can show carries the app in `topup_app`.
2026-11-13 23:00:39 -07:00
Nariman JelvehandClaude Opus 5.5 3e11e7dc79 refactor: keep folder sort and layout in puter.kv, drop /set_sort_by and /set_layout (#4642)
Explorer saved each folder's sort and layout through POST /set_sort_by and
/set_layout, which wrote them onto the folder's fsentries row. stat only
returns `layout`, so a saved sort was never read back.

The GUI now keeps them in the user's key-value store, one key per folder
uid (`folder_view:<uid>` holding `{ sort_by, sort_order, layout }`), via
helpers/folderViewPrefs.js. A change to one preference keeps the others.
A folder with nothing stored falls back to the layout its fsentry still
carries, then to the defaults, so existing layouts carry over.

refresh_item_container reads the folder's view after stat and re-sorts
items that went in under the previous sort; a view for a folder the
window has left is ignored. update_window_path no longer applies sort and
layout itself, since the refresh it starts does.

With no callers left, DesktopController and its two routes are removed.
The fsentries columns and stat's `layout` field stay.

Co-authored-by: Claude Opus 5.5 <[email protected]>
2026-11-13 21:34:37 -07:00
Nariman JelvehandClaude Opus 5.5 5b1b5b8a7a fix(puter-js): apps launched from desktop.puter.com take its messages (#4640)
In an app, puter.js trusted only https://puter.com as the GUI origin, so
every desktop message from desktop.puter.com failed the origin check and
was dropped. The desktop waits for the app's answer to windowWillClose,
so app windows there never closed; itemsOpened, drag-and-drop, focus,
theme and locale changes were lost the same way, and a reauth request
was posted to the wrong origin.

An app now takes its GUI origin from the launch URL's `puter.origin`,
but only when that is one of Puter's own desktops (exactly
https://puter.com or https://desktop.puter.com); anything else keeps
https://puter.com, so a page that frames an app still can't drive it.
An explicit PUTER_ORIGIN still wins, and a top-level page never reads it.

Co-authored-by: Claude Opus 5.5 <[email protected]>
2026-11-13 20:51:33 -07:00
Daniel Salazar b2eb8ed895 feat(auth): a first-party page served on its own origin signs in with a session of its own (#4635)
A deployment can list origins of its own pages in
`first_party_session_origins`. When such a page asks the GUI to sign it in,
the sign-in hands it a GUI token on a `web` session of its own, parented to
the user's sign-in, instead of an app token.

The session is minted by POST /auth/first-party-session, which only the GUI's
own page script may call, and only as the user's sign-in itself: app tokens,
access tokens and a session minted this way are refused. It never gets a
session cookie, ends with the sign-in it was handed out under (revoking either
takes the tokens minted under the page along), is not counted as a sign-in,
and stays up with its sign-in when all other sessions are revoked.

The sessions list shows it under its sign-in, titled with the page's host and
badged as a site session.
2026-11-13 17:47:57 -07:00
Daniel Salazar d1cbfb9fb1 fix(puter-js): a godmode app opened in its own tab stays signed in across a reload (#4632)
A tab the desktop opened for an app boots the SDK as a top-level page, where
a session-type token was kept in memory only. Once the desktop hands such a
tab its launch token, the SDK now keeps it in sessionStorage, as a framed app
does, and a reload of that tab reads it back before any stored app token.

A page that wasn't handed a token this way still persists no session token.
2026-11-13 15:46:17 -07:00
Neal ShahandClaude Opus 5.5 ac44de819d fix(hosting): choose the site and subdomain from Host, not X-Forwarded-Host (#4459)
With trust proxy on, express derives req.hostname and req.subdomains from
X-Forwarded-Host, which any client can send, while host validation reads the
raw Host. A request could serve one hosted site's files on the main domain,
or reach the API on a hosting host. The header is now dropped before routing.

Fixes PUT-2272

Co-authored-by: Claude Opus 5.5 <[email protected]>
2026-11-13 14:56:16 -05:00
404oops f044d8d071 Restore the CODEOWNERS default owner and wildcard patterns
The default owner line read `- @Salazareo`, which only matches a file
named `-`, so files without a specific owner had none. Three patterns
had their `*` escaped, which GitHub rejects as invalid, so their lines
assigned no owner.
2026-11-13 07:52:52 +01:00
404oops aea6e662f6 Fuzz the AI drivers with seeded requests and upstream answers
Two seeded suites send malformed input through the real drivers and
providers. The vendors are played at the HTTP boundary, so each
provider's request and response code runs.

- ChatCompletionDriver.fuzz.test.ts covers the 17 chat providers that
  have a static catalog. Requests with a few values changed must
  resolve or reject below 500 without marking a route unhealthy.
  Valid requests against wrong statuses, bodies of the wrong shape and
  broken event streams must end with an answer or a classified error.
  A stream only has to end; the error line it carries is not checked.
- mediaDrivers.fuzz.test.ts covers the image, video, speech and OCR
  drivers, called as a user with unlimited metering so a request gets
  past file input and the credit hold. Every vendor refuses the
  request, and every call must end with a classified error.

shared/fuzzTestUtil.ts holds the seeded random source, the hostile JSON
values, the mutation, and the stand-in for fetch, undici and axios. An
SDK that calls https.request itself fails the suite. PUTER_FUZZ_CASES
raises the case count for a longer run. doc/ai-drivers.md describes the
suites.
2026-11-13 07:52:52 +01:00
404oops 31f6dc2d37 Answer malformed AI requests with a 400 where they enter
A value that used to end in an unclassified error, in a request or in
an upstream answer, now gets a classified one before provider code runs
on it. A value a driver or provider ignored or refused before is
handled as it was.

Every driver:
- shared/scalarOptions.ts answers 400 for a value that cannot be
  converted to text, such as { "toString": null }, in the options a
  driver reads as text, a number or a flag. The chat, video, text to
  speech, speech to text, voice changer and OCR drivers call it first.
- An unknown provider name that is not text gets the same 400 as any
  other unknown name.

Chat requests:
- A falsy `tools` means no tools on every provider.
- A tool call id or a content block type that cannot be converted to
  text is a 400.
- A tool result that is an object is sent as its JSON, like the content
  of a `tool` message.
- `include` that is not an array is a 400.
- `anthropicBetas` that is not a list of names is dropped.
- Gemini's native route answers 400 when no user or assistant message
  is left to send.
- The prompt estimate reads a compaction block whose content is not a
  string.

Image, video, speech and OCR requests:
- The image driver checks `input_image_mime_type` with `quality` and
  `resolution`.
- The video driver answers 400 for a model that is not a string.
- The voice changer answers 400 for an `output_format` that is not a
  string.
- Mistral OCR answers 400 when the SDK refuses the options before
  sending them, as Mistral chat does.

Upstream answers to chat:
- A choice without a message is the existing 400 bad_response, which
  still falls back.
- A usage count that is not a number is reported as zero in the cost
  event, as pricing already skips it.
2026-11-13 07:52:52 +01:00
404oops 2b03bc53b1 Rerun Replicate predictions an output safety checker rejects
Some community models check the generated image and fail the
prediction with "Try running it again" for a share of seeds, whatever
the prompt. adirik/realvisxl-v3.0-turbo failed 5 of 24 runs of "a small
red cube on a white table" that way.

A prediction that fails with that message is now run again, up to two
more times, inside the same deadline and credit hold. A request that
pins a seed is not rerun, because the same seed is rejected again.
Only the prediction that succeeds is metered, as before.
2026-11-13 07:52:52 +01:00
404oops 3ea0010220 Remove the unused ImageProvider base class
Every image provider implements IImageProvider directly; nothing extended
or imported the abstract class in interfaces/image/ImageProvider.ts.

The guide's import rules now say that only an interface's driver file
imports providers, which holds since the tool helpers stopped importing
the Claude policy.
2026-11-13 07:52:52 +01:00
404oops 395e4bd54a Split the tool and compaction helpers by dialect
FunctionCalling.js and compaction.js each held Puter-facing parsing next
to OpenAI and Anthropic request shapes. Each dialect now owns its mapping:

  interfaces/chat/utils     normalize_tools_object, readCompaction and the
                            other dialect-neutral checks
  protocols/openaiCompatible/openaiParams.ts
                            toolsToWire (was make_openai_tools) and
                            toOpenAiContextManagement
  providers/claude/chat/contextManagement.ts
                            toAnthropicContextManagement

toolsToWire takes the dialect as its second argument, like the other
mappers in that module, and takes an array; every caller already guarded
the empty case.

make_claude_tools is removed. Nothing but its tests called it: the Claude
provider uses claudeToolPolicy directly. The three cases only those tests
had are added to the claudeToolPolicy tests. The chat interface helpers no
longer import from providers/claude.
2026-11-13 07:52:52 +01:00
404oops b55cb715ca Move the AI test helpers into the AI tree
drivers/integrationTestUtil.ts held helpers for AI provider tests, but a
store test and the realtime relay test imported from it too.

  drivers/ai/shared/integrationTestUtil.ts   the provider test helpers
  testEnv.ts                                 optionalEnv and skipUnlessEnv,
                                             for any env-gated suite
  testUtil.ts                                withAiCostFactorHook, next to
                                             the server it hooks

No test under services/ or stores/ imports from the driver layer's AI
code now. The typecheck baseline entry for the moved file is renamed, not
regenerated.
2026-11-13 07:52:52 +01:00
404oops 2e4d918900 Remove the aiUserIdentifier alias
shared/aiUserIdentifier.ts re-exported upstreamUserIdentifier under a
second name, and only its own test imported it. The alias is gone; its
tests, which cover truncation and actor shapes the other file didn't,
replace the four cases in upstreamIdentifier.test.ts.
2026-11-13 07:33:27 +01:00
404oops 8c799a433a Move the AI billing helpers the realtime relay uses below the driver layer
OpenAIRealtimeRelayService is a service, and it imported the cost factor,
the provider key reader and a transcription price from drivers/ai. Those
now live in src/backend/util, which both layers can import:

  util/aiCostFactor.ts    withAiCostFactor, withAiCreditHold and the
                          request cost helpers, unchanged
  util/aiProviderKey.ts   readProviderKey, cut from providerRegistry.ts
  util/aiPrices.ts        the gpt-transcribe per-second price, read by the
                          OpenAI speech-to-text cost table and the relay

withStreamedAiCharge returns a DriverStreamResult, so it stays in the
driver layer as drivers/ai/shared/streamedAiCharge.ts with its tests.

MeteringService is untouched. No prices or billing behavior change.
2026-11-13 07:33:27 +01:00
404oops 5f84d5897b Move the ElevenLabs voice changer into a provider class
VoiceChangerDriver built the ElevenLabs request, measured the converted
audio and metered it, and imported the output formats from the ElevenLabs
text-to-speech provider. That code is now in providers/elevenlabs:

  outputFormats.ts                         formats both speech interfaces use
  speechToSpeech/ElevenLabsSpeechToSpeechProvider.ts
  speechToSpeech/measuredOutput.ts         WAV, Opus and byte-rate timing
  speechToSpeech/costs.ts

The driver keeps provider resolution, test mode and the auth check. With
one provider it holds the class directly; there is no provider interface.

No behavior change: same validation order, error messages, usage keys,
holds and charges.
2026-11-13 07:33:27 +01:00
404oops c0aff857a4 Move the Textract and Mistral OCR calls into provider classes
OCRDriver held both vendors' clients, request building, response
normalizing and metering. That now lives in TextractOcrProvider
(providers/awsTextract/ocr) and MistralOcrProvider (providers/mistral/ocr),
both implementing IOcrProvider. The driver keeps the provider catalog,
model resolution, auth and credit gates, and input resolution.

The page estimate, document detection and credit hold both providers use
are in interfaces/ocr/common.ts.

No behavior change: same validation order, error messages, usage keys,
holds and charges.
2026-11-13 07:33:26 +01:00
404oops ff606abf3c Move catalogPrice into ai/shared
Chat providers and the Cloudflare and xAI image catalogs all parse vendor
prices with catalogPrice, so it lives in shared/ instead of the chat
interface's pricing helpers. The image catalogs no longer import from
the chat interface.
2026-11-13 07:33:26 +01:00
404oops 8159998830 Document the AI driver layout
doc/ai-drivers.md explains what interfaces, providers, protocols and
shared each own, the import rules between them, where a new file goes,
and the steps for adding a provider or an AI interface. It also lists
the three places that don't follow the layout yet.

doc/architecture.md and AGENTS.md link to it.
2026-11-13 07:33:26 +01:00
404oops 395f7f6adb Move the AI drivers into drivers/ai
The seven drivers/ai-* folders and the AI-only helpers in drivers/util
become one tree:

  drivers/ai/interfaces/<interface>/            the Puter-facing drivers
  drivers/ai/providers/<vendor>/<interface>/    vendor adapters, catalogs, tests
  drivers/ai/protocols/openaiCompatible/        the shared OpenAI wire protocol
  drivers/ai/shared/                            helpers several interfaces use

Files are moved and their relative imports rewritten; no implementation
changes. Driver names, aliases, interfaces, config keys, model ids and
usage keys are unchanged.

The chat catalog registration test scanned providers/*/ next to itself.
It now scans providers/*/chat/ and fails if the scan finds no catalog, so
a wrong directory can't pass it.

General helpers that non-AI code also uses stay in drivers/util: file
input, data URIs, upstream errors, stream piping, CRUD limits.
2026-11-13 07:33:26 +01:00
401 changed files with 8297 additions and 4815 deletions

No files matched your search

+8 -5
View File
@@ -2,7 +2,7 @@
# The last matching rule takes precedence.
- @Salazareo
* @Salazareo
# GUI
@@ -18,7 +18,10 @@
# Backend AI
/src/backend/controllers/puterai/ @404oops @ProgrammerIn-wonderland @Salazareo
/src/backend/drivers/ai-\*/ @404oops @ProgrammerIn-wonderland @Salazareo
/src/backend/drivers/ai/ @404oops @ProgrammerIn-wonderland @Salazareo
/src/backend/util/aiCostFactor* @404oops @ProgrammerIn-wonderland @Salazareo
/src/backend/util/aiPrices* @404oops @ProgrammerIn-wonderland @Salazareo
/src/backend/util/aiProviderKey* @404oops @ProgrammerIn-wonderland @Salazareo
# Backend: AI, hosting, workers, networking, and WebDAV
@@ -29,7 +32,7 @@
/src/backend/drivers/workers/ @ProgrammerIn-wonderland @Salazareo
/src/backend/services/localworker/ @ProgrammerIn-wonderland @Salazareo
/src/backend/stores/subdomain/ @ProgrammerIn-wonderland @Salazareo
/extensions/workerSandbox\* @ProgrammerIn-wonderland @Salazareo
/extensions/workerSandbox* @ProgrammerIn-wonderland @Salazareo
/src/worker/ @ProgrammerIn-wonderland @Salazareo
/src/worker-types/ @ProgrammerIn-wonderland @Salazareo
/src/backend/controllers/peer/ @ProgrammerIn-wonderland @Salazareo
@@ -42,7 +45,7 @@
/src/backend/services/permission/ @jfcastro92 @Salazareo
/src/backend/services/acl/ @jfcastro92 @Salazareo
/src/backend/services/team/ @jfcastro92 @Salazareo
/src/backend/services/apps/AppPermissionService\* @jfcastro92 @Salazareo
/src/backend/services/apps/AppPermissionService* @jfcastro92 @Salazareo
/src/backend/stores/share/ @jfcastro92 @Salazareo
/src/backend/stores/permission/ @jfcastro92 @Salazareo
/src/backend/stores/group/ @jfcastro92 @Salazareo
@@ -59,7 +62,7 @@
/src/puter-js/tests/api/suites/hosting.suite.ts @ProgrammerIn-wonderland @Salazareo
/src/puter-js/tests/api/suites/workers.suite.ts @ProgrammerIn-wonderland @Salazareo
/src/puter-js/tests/api/suites/net.suite.ts @ProgrammerIn-wonderland @Salazareo
/src/puter-js/src/modules/Peer\* @ProgrammerIn-wonderland @Salazareo
/src/puter-js/src/modules/Peer* @ProgrammerIn-wonderland @Salazareo
# SDK: sharing and permissions
+6 -1
View File
@@ -9,6 +9,7 @@ Use these as the source of truth before exploring further:
- [README.md](README.md) — project overview and quickstart.
- [doc/architecture.md](doc/architecture.md) — backend layered stack (controllers → drivers → services → stores → clients), `PuterServer` wiring, `Context` (ALS), actors, and extensions.
- [doc/contributing-apis.md](doc/contributing-apis.md) — adding and maintaining public APIs end to end (backend surface → puter.js → types → docs → tests). Follow it for any API work.
- [doc/ai-drivers.md](doc/ai-drivers.md) — where AI driver code lives (interfaces, providers, protocols, shared), the import rules between those folders, and how to add a provider or an AI interface.
- [doc/pagination.md](doc/pagination.md) — the one pagination convention for list APIs (limit/cursor/offset/includeTotal, envelope shape, cursor semantics).
- [doc/alarms.md](doc/alarms.md) — raising alarms and picking a severity (what pages, what only gets recorded), plus the config that routes them.
- [doc/self-hosting.md](doc/self-hosting.md) — running Puter outside hosted infra.
@@ -70,7 +71,7 @@ Commit messages and PR descriptions never credit anyone or mention a bug report.
- **Test new behavior.** Every new function, endpoint, driver method, or logic branch gets a test; every bug fix gets a regression test that fails before the fix. If something is genuinely hard to test, skip it but say so in the PR.
- **Boy Scout Rule, proportional to the change.** In files you're already touching, fix the typo, the dead import, the missing test, the bit you had to read twice. Don't ride a refactor along with a bug fix.
- **Understand what you commit.** AI assistance is fine; shipping code you couldn't have written, debugged, or defended in review is not.
- **A limit change is not done until the docs change with it.** Every rate limit, concurrency cap, quota, and allowance in the code is published in [src/docs/src/rate-limits-and-quotas.md](src/docs/src/rate-limits-and-quotas.md) — an undisclosed limit is one developers discover as a service failure. If a PR moves any of these numbers, the same PR updates that page. The numbers live in `src/backend/controllers/fs/limits.ts` (filesystem), `src/backend/drivers/util/aiLimits.ts` (all AI drivers), per-driver `rateLimit`/`concurrent` configs, and `src/backend/data/subPolicies/` (free-tier allowances).
- **A limit change is not done until the docs change with it.** Every rate limit, concurrency cap, quota, and allowance in the code is published in [src/docs/src/rate-limits-and-quotas.md](src/docs/src/rate-limits-and-quotas.md) — an undisclosed limit is one developers discover as a service failure. If a PR moves any of these numbers, the same PR updates that page. The numbers live in `src/backend/controllers/fs/limits.ts` (filesystem), `src/backend/drivers/ai/shared/aiLimits.ts` (all AI drivers), per-driver `rateLimit`/`concurrent` configs, and `src/backend/data/subPolicies/` (free-tier allowances).
---
@@ -97,6 +98,10 @@ Layer rules:
- **Drivers and controllers stay thin.** Domain rules a second entry point needs (signup, app CRUD, worker deploy, AI routing/billing) live in a service, not in the first driver or controller that needed them.
- **Prefer explicit arguments over [`Context`](src/backend/core/context.ts) (ALS).** Use it only for request-scoped values that would otherwise thread through many layers; today that's mostly `actor` and `req`.
### AI drivers
AI drivers live in [src/backend/drivers/ai/](src/backend/drivers/ai/): `interfaces/` holds the Puter-facing drivers, `providers/<vendor>/<interface>/` holds vendor code, `protocols/` holds wire protocols several vendors speak, and `shared/` holds helpers more than one interface uses. Follow [doc/ai-drivers.md](doc/ai-drivers.md) when adding or moving AI code.
### Extensions
[extensions/](extensions/) is for **non-crucial parts of the system**: Puter must still work with the extension removed. If core needs to call it, it belongs in core ([whoami](extensions/whoami.ts) is the cautionary example). Inside an extension, follow the same layered structure, unless it only needs a few route handlers, in which case the `extension.get/post/...` helpers are enough on their own. The `extension` API is in [doc/architecture.md](doc/architecture.md#extension-api).
+152
View File
@@ -0,0 +1,152 @@
# AI Drivers
Where the AI driver code lives under [src/backend/drivers/ai/](../src/backend/drivers/ai/), what each folder owns, and how to add a provider or a new AI interface. The general layer rules are in [architecture.md](architecture.md) and [AGENTS.md](../AGENTS.md#backend). This page covers what is specific to AI.
## Layout
```text
src/backend/drivers/ai/
├── interfaces/ Puter-facing drivers, one folder per AI interface
├── providers/ Vendor code, as providers/<vendor>/<interface>/
├── protocols/ Wire protocols that several vendors speak
└── shared/ Helpers that more than one interface uses
```
| Folder | Owns | Example |
| --- | --- | --- |
| `interfaces/<interface>/` | The driver class, the interface's contracts in `types.ts`, its provider base class where one exists, and the helpers and tests that apply to every provider of that interface. | [interfaces/chat/ChatCompletionDriver.ts](../src/backend/drivers/ai/interfaces/chat/ChatCompletionDriver.ts) |
| `providers/<vendor>/<interface>/` | One vendor's implementation of one interface: the adapter, its model catalog and prices, its fixtures, and its tests. | [providers/gemini/video/](../src/backend/drivers/ai/providers/gemini/video/) |
| `providers/<vendor>/` | Helpers that several of that vendor's interfaces use. | [providers/gemini/geminiRest.ts](../src/backend/drivers/ai/providers/gemini/geminiRest.ts) |
| `protocols/openaiCompatible/` | The OpenAI wire protocol: the base provider class, message and usage conversion, and parameter mapping for Chat Completions and Responses. | [protocols/openaiCompatible/OpenAICompatProvider.ts](../src/backend/drivers/ai/protocols/openaiCompatible/OpenAICompatProvider.ts) |
| `shared/` | Helpers that more than one interface uses: limits, the provider registry, the remote catalog cache, job polling, streamed charging, and media input. | [shared/aiLimits.ts](../src/backend/drivers/ai/shared/aiLimits.ts) |
Helpers that non-AI code also uses stay in [src/backend/drivers/util/](../src/backend/drivers/util/): file input, data URIs, upstream error mapping, and stream piping.
AI code that a service also needs lives in [src/backend/util/](../src/backend/util/), because a service can't import from the driver layer. The realtime relay in `services/realtime/` bills through the same cost factor as the drivers, so these three files are there:
| File | Holds |
| --- | --- |
| [aiCostFactor.ts](../src/backend/util/aiCostFactor.ts) | `withAiCostFactor`, the metering view every AI driver and the realtime relay record usage through, and `withAiCreditHold`. |
| [aiProviderKey.ts](../src/backend/util/aiProviderKey.ts) | `readProviderKey`, which reads an API key from `config.providers` blocks. |
| [aiPrices.ts](../src/backend/util/aiPrices.ts) | Vendor prices that a driver and a service both bill. |
### Interfaces
| Folder | Driver | `driverInterface` | `driverName` | Key in `puterDrivers` |
| --- | --- | --- | --- | --- |
| `interfaces/chat/` | `ChatCompletionDriver` | `puter-chat-completion` | `ai-chat` | `aiChat` |
| `interfaces/image/` | `ImageGenerationDriver` | `puter-image-generation` | `ai-image` | `aiImage` |
| `interfaces/video/` | `VideoGenerationDriver` | `puter-video-generation` | `ai-video` | `aiVideo` |
| `interfaces/textToSpeech/` | `TTSDriver` | `puter-tts` | `ai-tts` | `aiTts` |
| `interfaces/speechToText/` | `SpeechToTextDriver` | `puter-speech2txt` | `ai-speech2txt` | `aiSpeech2Txt` |
| `interfaces/speechToSpeech/` | `VoiceChangerDriver` | `puter-speech2speech` | `ai-speech2speech` | `aiSpeech2Speech` |
| `interfaces/ocr/` | `OCRDriver` | `puter-ocr` | `ai-ocr` | `aiOcr` |
Folder names are not runtime identifiers. `driverInterface`, `driverName`, driver aliases, the keys under `providers` in config, provider ids, model ids, and usage keys are public contracts. Moving or renaming a file never changes them.
## Request flow
A call goes from the interface to a provider, and from there to a shared protocol when the vendor speaks one:
```text
interfaces/chat/ChatCompletionDriver.ts
validates the call, picks the model and provider, holds credits
→ providers/deepseek/chat/DeepSeekProvider.ts
DeepSeek's catalog, endpoint, and request differences
→ protocols/openaiCompatible/OpenAICompatProvider.ts
the OpenAI-compatible request, response, and stream handling
```
A provider whose vendor has its own API, such as `ClaudeProvider` or `ElevenLabsTTSProvider`, implements the interface's contract directly and uses no protocol.
## Import rules
- `shared/` imports nothing from `interfaces/`, `providers/`, or `protocols/`.
- Nothing in `services/`, `stores/`, or `clients/` imports from `drivers/ai/`. Code that a service and a driver both need goes in `src/backend/util/`.
- `providers/` and `protocols/` import an interface's `types.ts` and helpers. They never import a driver class. Their tests can, to drive a provider through the real driver.
- Only an interface's driver file imports providers, to register them. The interface's helpers don't.
- An interface doesn't import another interface. When a second interface needs a helper, move the helper to `shared/`, or to `providers/<vendor>/` when it belongs to one vendor.
- A vendor folder imports another vendor's folder only when it serves that vendor's models. `providers/azure/` reads the OpenAI and xAI catalogs for that reason.
## Where a file goes
| The code is | It goes in |
| --- | --- |
| Input validation, gates, routing, credit holds, or anything else every provider of the interface gets | `interfaces/<interface>/` |
| A request or response contract that providers implement | `interfaces/<interface>/types.ts` |
| A vendor's endpoint, request shape, catalog, prices, or quirks | `providers/<vendor>/<interface>/` |
| A vendor's auth, REST plumbing, or constants that two of its interfaces use | `providers/<vendor>/` |
| Behavior of a wire protocol that several vendors speak | `protocols/<protocol>/` |
| A helper that two or more interfaces use and that belongs to no vendor | `shared/` |
| A helper that non-AI drivers or controllers also use | `src/backend/drivers/util/` |
| AI code that a service also uses | `src/backend/util/`, in a file named `ai<Something>.ts` |
Keep a helper next to its only consumer. Move it up when a second consumer appears, not before.
When one Puter option becomes a different request shape per vendor, the interface reads the option and each dialect maps it. The chat `compaction` option is read in `interfaces/chat/utils/compaction.js`, mapped for OpenAI by `toOpenAiContextManagement` in `protocols/openaiCompatible/openaiParams.ts`, and mapped for Anthropic by `toAnthropicContextManagement` in `providers/claude/chat/contextManagement.ts`. Tools work the same way: `normalize_tools_object` in the interface, then `toolsToWire` for OpenAI and `claudeToolPolicy` for Anthropic.
Tests sit next to the code they test. Fixtures go in a `fixtures/` folder next to the test that reads them.
## Adding a provider
1. Create `providers/<vendor>/<interface>/`. Use the vendor's existing folder if it has one, so all of a vendor's interfaces stay together.
2. Implement the interface's contract from `interfaces/<interface>/types.ts`:
| Interface | Contract | Base class |
| --- | --- | --- |
| chat | `IChatProvider` | `OpenAICompatProvider` for an OpenAI-compatible vendor |
| image | `IImageProvider` | none |
| video | `IVideoProvider` | `VideoProvider` |
| textToSpeech | `ITTSProvider` | none |
| speechToText | `ISpeechToTextProvider` | `SpeechToTextProvider` |
| ocr | `IOcrProvider` | none |
Speech to speech has one provider, so `VoiceChangerDriver` uses `ElevenLabsSpeechToSpeechProvider` directly. Add a contract to its `types.ts` when a second provider needs one.
3. Put the model catalog and prices in the same folder, as `models.ts` or `costs.ts`. A provider that loads its catalog from the vendor wraps the fetch in `cachedRemoteCatalog` from `shared/`.
4. Register the provider in the interface's driver: in `#registerProviders()`, or in `onServerStart()` for OCR and speech to speech. A chat provider built from an API key and an optional base URL is one row in `KEYED_PROVIDERS` instead. Text to speech and speech to text also list the provider id and its aliases in `providerAliases.ts`.
5. Read credentials from `config.providers[<key>]`. The key is part of the self-hosting config contract, so reuse the vendor's existing key when it has one.
6. Add tests next to the provider. Mock the vendor SDK or `fetch` at the network boundary and drive the provider through the real driver where you can. A live test is a `*.integration.test.ts` file that skips itself, with `skipUnlessEnv` from [testEnv.ts](../src/backend/testEnv.ts), unless its `PUTER_TEST_AI_*` variable is set. Helpers for provider tests are in [shared/integrationTestUtil.ts](../src/backend/drivers/ai/shared/integrationTestUtil.ts).
7. For a chat provider, add its static catalog to `CATALOGS` in [modelCatalogs.test.ts](../src/backend/drivers/ai/interfaces/chat/modelCatalogs.test.ts) and the provider to [providerConsistency.test.ts](../src/backend/drivers/ai/interfaces/chat/providerConsistency.test.ts). The catalog test fails when a `providers/*/chat/` file with "model" in its name exports a catalog that isn't listed.
8. Add the provider's config key to the `providers` config of the interface's [fuzz suite](#fuzz-tests), and its id to the list of providers the suite expects.
9. Update the published docs for the models and limits the provider adds, following [contributing-apis.md](contributing-apis.md).
### OpenAI-compatible vendors
A vendor that speaks the OpenAI Chat Completions protocol extends `OpenAICompatProvider` and overrides only what differs. It still gets its own `providers/<vendor>/chat/` folder.
Vendors share one constants-only folder only when at least six of them differ in nothing but base URL and model catalog. Each then becomes one data file under `providers/openaiCompatible/providers/`, read by a single implementation. No vendor qualifies today, because every subclass overrides request, catalog, or billing behavior.
## Fuzz tests
Two seeded suites send malformed input through the real drivers and providers. They run with the rest of the backend tests.
| Suite | Covers |
| --- | --- |
| [interfaces/chat/ChatCompletionDriver.fuzz.test.ts](../src/backend/drivers/ai/interfaces/chat/ChatCompletionDriver.fuzz.test.ts) | Chat requests with a few values changed, and valid requests against wrong statuses, bodies of the wrong shape, and broken event streams. |
| [interfaces/mediaDrivers.fuzz.test.ts](../src/backend/drivers/ai/interfaces/mediaDrivers.fuzz.test.ts) | Image, video, speech and OCR requests with a few values changed. Every vendor refuses the request. |
What the suites assert:
- A malformed request never fails with an error that has no status, which `/drivers/call` would report as an unclassified 500.
- For chat, with the vendors answering, a malformed request also never fails with a 5xx and never marks a route unhealthy.
- When the upstream misbehaves, chat ends with an answer or an `HttpError`. A stream only has to end: the error line a stream carries isn't checked, so a malformed event can still end one with a 500.
The vendors are played at the HTTP boundary by `upstream` in [shared/fuzzTestUtil.ts](../src/backend/drivers/ai/shared/fuzzTestUtil.ts), which stands in for global `fetch`, undici `fetch` and axios, and makes `https.request` throw so an SDK with its own client fails the suite. The vendors' SDKs run, except Groq's, which is swapped for the OpenAI SDK, and the AWS clients, whose `send` is stubbed. The media suite calls the drivers as a user with unlimited metering, so a request gets past file input and the credit hold to the vendor. A provider that loads its catalog from the vendor registers no models there, so the suites skip it.
A failure lists each distinct finding with its case number and input. The random source is seeded, so the same case fails again on the next run. To search further, raise the case count:
```sh
PUTER_FUZZ_CASES=100000 npx vitest run --config src/backend/vitest.config.ts src/backend/drivers/ai/interfaces/chat/ChatCompletionDriver.fuzz.test.ts
```
When a suite finds a crash, validate the value where the request enters, in the driver or in `normalize_single_message`, so every provider is covered. Reject only what crashed: a value a provider ignored or refused before must still reach it. An option a driver reads as text, a number or a flag goes in the driver's `SCALAR_OPTIONS`, and `assertScalarOptions` from `shared/scalarOptions.ts` answers 400 for a value there that can't be converted to text. Add a unit test for the case next to the fix.
## Adding an interface
1. Create `interfaces/<interface>/` with the driver class and a `types.ts` for the contracts providers implement.
2. Register the driver in [src/backend/drivers/index.ts](../src/backend/drivers/index.ts).
3. Use `AI_RATE_LIMIT` and `AI_CONCURRENT` from `shared/aiLimits.ts`, and publish any new limit in [rate-limits-and-quotas.md](../src/docs/src/rate-limits-and-quotas.md).
4. Meter through `withAiCostFactor` from `src/backend/util/aiCostFactor.ts`, and route between providers with `ProviderRegistry` from `shared/providerRegistry.ts`.
5. List the options the providers read as text, a number or a flag in a `SCALAR_OPTIONS` array, and call `assertScalarOptions` from `shared/scalarOptions.ts` first in each method. Add the driver to [mediaDrivers.fuzz.test.ts](../src/backend/drivers/ai/interfaces/mediaDrivers.fuzz.test.ts).
6. Follow [contributing-apis.md](contributing-apis.md) for the puter.js method, types, docs, and tests.
+13
View File
@@ -42,6 +42,19 @@ block-beta
Extensions sit alongside this stack and can register into any layer; see [Extensions](#extensions).
### AI drivers
The AI drivers have their own layout under [src/backend/drivers/ai/](../src/backend/drivers/ai/):
| Folder | Holds |
| --- | --- |
| `interfaces/` | The Puter-facing drivers, one folder per AI interface (chat, image, video, text to speech, speech to text, speech to speech, OCR). |
| `providers/` | Vendor code, as `providers/<vendor>/<interface>/`. |
| `protocols/` | Wire protocols that several vendors speak, such as the OpenAI-compatible one. |
| `shared/` | Helpers that more than one interface uses. |
A call goes from an interface's driver to a provider, and from there to a shared protocol when the vendor speaks one. [ai-drivers.md](ai-drivers.md) has the ownership rules, the import rules, and the steps for adding a provider.
## Entry point: `PuterServer`
`PuterServer` is the bootstrap. It:
@@ -21,7 +21,6 @@ import {
isAccessTokenActor,
isAccountContext,
isAppActor,
viaWorkerOf,
} from '../../core/actor.js';
import { HttpError } from '../../core/http/HttpError.js';
import {
@@ -202,13 +201,6 @@ export class AppController extends PuterController {
},
async (req, res) => {
const actor = req.actor;
// Server-side code running as the account (a worker, an events
// handler) is not someone opening the app. Answered as success
// so deployed workers, which still ask, don't log a failure.
if (viaWorkerOf(actor)) {
res.json({});
return;
}
const bodyAppUid = req.body?.app_uid;
const callerAppUid = actor?.effectiveApp?.uid;
const app_uid =
@@ -1143,34 +1143,6 @@ describe('AppController POST /rao actor gating', () => {
).rejects.toMatchObject({ statusCode: 403, legacyCode: 'forbidden' });
});
it('records nothing for a worker running as the app', async () => {
const owner = await makeUser();
const app = await createApp(owner.actor);
const workerActor = makeActor({
...owner.actor,
app: { uid: app.uid as string },
session: { uid: uuidv4(), kind: 'worker' },
} as unknown as Actor);
const { res, captured } = makeRes();
await withActor(workerActor, () =>
callRoute(
'post',
'/rao',
makeReq({ body: {}, actor: workerActor }),
res,
),
);
expect(captured.body).toEqual({});
await server.controllers.apps.drainPendingAppOpens();
const rows = (await server.clients.db.read(
'SELECT `app_uid` FROM `app_opens` WHERE `app_uid` = ? AND `user_id` = ?',
[app.uid, owner.userId],
)) as unknown[];
expect(rows).toHaveLength(0);
});
it("refuses to let an app actor record another app's open", async () => {
const { actor } = await makeUser();
const own = await createApp(actor);
@@ -4487,6 +4487,34 @@ export class AuthController extends PuterController {
res.json({ token: guiToken });
}
@Post('/auth/first-party-session', {
subdomain: 'api',
rateLimit: SESSION_LIMIT,
// Hands out a session, so only the GUI's own page script may ask, and
// only as the user's sign-in itself.
guiOriginOnly: true,
requireUserActor: true,
})
async handleFirstPartySession(req: Request, res: Response): Promise<void> {
const { origin } = req.body ?? {};
if (typeof origin !== 'string' || !origin) {
throw new HttpError(400, 'Missing `origin`', {
legacyCode: 'bad_request',
});
}
const token = await this.services.auth.createFirstPartySessionToken(
req.actor!,
origin,
{
ip: req.ip || req.socket?.remoteAddress,
user_agent: req.headers?.['user-agent'],
origin: req.headers?.origin,
host: req.headers?.host,
},
);
res.json({ token });
}
@Get('/session/sync-cookie', {
rateLimit: SESSION_LIMIT,
// Installs the session cookie. Only page script on our own origin
@@ -4511,6 +4539,14 @@ export class AuthController extends PuterController {
res.status(400).end();
return;
}
// A session handed to a first-party page never gets a cookie.
if (req.actor.session.parentUid) {
throw new HttpError(
403,
'This session cannot install a session cookie',
{ legacyCode: 'forbidden' },
);
}
// As in `/get-gui-token`: the actor is the user's row as auth read it.
const sessionToken = this.services.auth.createSessionTokenForSession(
req.actor.user as UserRow,
@@ -0,0 +1,282 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { afterAll, beforeAll, describe, expect, it } from 'vitest';
import type { IConfig } from '../../types.js';
import { setupPuterTestEnv, type PuterTestEnv } from '../../testUtil.js';
const PAGE = 'https://desk.first-party.test';
// Seeded with godmode on.
const DEV_CENTER = 'app-0b37f054-07d4-4627-8765-11bd23e889d4';
/**
* One of the deployment's own pages, served apart from the GUI, signs in with a
* `web` session of its own under the user's sign-in.
*/
describe('first-party page sessions', () => {
let env: PuterTestEnv;
let userId = 0;
beforeAll(async () => {
env = await setupPuterTestEnv({
first_party_session_origins: [`${PAGE}/`],
} as IConfig);
const row = await env.server.stores.user.getByUsername(
env.users.user.username,
);
userId = row!.id;
}, 120_000);
afterAll(async () => {
await env?.shutdown();
});
const call = (
method: string,
path: string,
token: string,
body?: unknown,
headers: Record<string, string> = {},
) =>
fetch(new URL(path, env.apiOrigin), {
method,
headers: {
'content-type': 'application/json',
authorization: `Bearer ${token}`,
...headers,
},
...(body === undefined ? {} : { body: JSON.stringify(body) }),
});
/** A sign-in of its own, so revoking it leaves the shared one up. */
const signIn = async () => {
const user = await env.server.stores.user.getById(userId);
const { gui_token, session } =
await env.server.services.auth.createSessionToken(user!);
return { token: gui_token, uuid: session.uuid as string };
};
const mint = async (token: string, origin = PAGE) => {
const res = await call('POST', '/auth/first-party-session', token, {
origin,
});
expect(res.status).toBe(200);
return ((await res.json()) as { token: string }).token;
};
const actorOf = async (token: string) =>
(await env.server.services.auth.authenticate(token)).actor;
const sessions = async (token: string) => {
const res = await call('GET', '/auth/list-sessions', token);
expect(res.status).toBe(200);
return (await res.json()) as Array<Record<string, unknown>>;
};
it('hands the page a session of its own under the sign-in', async () => {
const root = await signIn();
const token = await mint(root.token);
const actor = await actorOf(token);
expect(actor?.app ?? null).toBeNull();
expect(actor?.accessToken ?? null).toBeNull();
expect(actor?.session).toMatchObject({
kind: 'web',
parentUid: root.uuid,
});
expect(actor?.session?.uid).not.toBe(root.uuid);
const row = (await sessions(root.token)).find(
(s) => s.uuid === actor!.session!.uid,
);
expect(row).toMatchObject({
kind: 'web',
parent_session_id: root.uuid,
first_party_origin: PAGE,
current: false,
});
});
it('reuses the session on a second ask', async () => {
const root = await signIn();
const first = await actorOf(await mint(root.token));
const second = await actorOf(await mint(root.token));
expect(second?.session?.uid).toBe(first?.session?.uid);
});
it('acts as the account: launches apps, godmode ones included', async () => {
const root = await signIn();
const token = await mint(root.token);
const res = await call('POST', '/auth/get-user-app-token', token, {
app_uid: DEV_CENTER,
});
expect(res.status).toBe(200);
expect(await res.json()).toMatchObject({ godmode: true });
});
it('refuses an origin that is not listed', async () => {
const root = await signIn();
const res = await call(
'POST',
'/auth/first-party-session',
root.token,
{
origin: 'https://elsewhere.test',
},
);
expect(res.status).toBe(403);
});
it('refuses a page on another origin, the listed page included', async () => {
const root = await signIn();
for (const origin of [PAGE, 'https://elsewhere.test']) {
const res = await call(
'POST',
'/auth/first-party-session',
root.token,
{ origin: PAGE },
{ origin },
);
expect(res.status).toBe(403);
}
});
it('refuses app tokens, access tokens and a first-party session', async () => {
const root = await signIn();
const page = await mint(root.token);
const app = (await (
await call('POST', '/auth/get-user-app-token', root.token, {
origin: 'https://some-app.test',
})
).json()) as { token: string };
const godmode = (await (
await call('POST', '/auth/get-user-app-token', root.token, {
app_uid: DEV_CENTER,
})
).json()) as { token: string };
for (const token of [page, app.token, godmode.token]) {
const res = await call('POST', '/auth/first-party-session', token, {
origin: PAGE,
});
expect(res.status).toBe(403);
}
});
it('never gets a session cookie', async () => {
const root = await signIn();
const page = await mint(root.token);
const syncCookie = (token: string) =>
fetch(new URL('/session/sync-cookie', env.origin), {
headers: { authorization: `Bearer ${token}` },
});
expect((await syncCookie(root.token)).status).toBe(204);
const res = await syncCookie(page);
expect(res.status).toBe(403);
expect(res.headers.get('set-cookie')).toBeNull();
});
it('ends with the sign-in it was handed out under', async () => {
const root = await signIn();
const page = await mint(root.token);
expect(await actorOf(page)).toBeDefined();
await env.server.services.auth.revokeSession(root.uuid);
const result = await env.server.services.auth.authenticate(page);
expect(result.actor).toBeUndefined();
expect(result.reauth?.reason).toBe('session_revoked');
});
it('takes the tokens minted under it along when its sign-in ends', async () => {
const root = await signIn();
const page = await mint(root.token);
const godmode = (await (
await call('POST', '/auth/get-user-app-token', page, {
app_uid: DEV_CENTER,
})
).json()) as { token: string };
const scoped = (await (
await call('POST', '/auth/create-access-token', godmode.token, {
permissions: ['service:foo:ii:read'],
expiresIn: '7d',
})
).json()) as { token: string };
expect(await actorOf(scoped.token)).toBeDefined();
await env.server.services.auth.revokeSession(root.uuid);
for (const token of [godmode.token, scoped.token]) {
expect(
(await env.server.services.auth.authenticate(token)).actor,
).toBeUndefined();
}
});
it('is revoked on its own without ending the sign-in', async () => {
const root = await signIn();
const page = await mint(root.token);
const pageUuid = (await actorOf(page))!.session!.uid;
await env.server.services.auth.revokeSession(pageUuid);
expect(
(await env.server.services.auth.authenticate(page)).actor,
).toBeUndefined();
expect(await actorOf(root.token)).toBeDefined();
});
it('keeps the whole current sign-in when revoking all others', async () => {
const root = await signIn();
const page = await mint(root.token);
const other = await signIn();
const otherPage = await mint(other.token);
// Asked from the page: its sign-in stays, and so does the page.
await env.server.services.auth.revokeAllSessions(
(await actorOf(page))!,
);
expect(await actorOf(page)).toBeDefined();
expect(await actorOf(root.token)).toBeDefined();
expect(
(await env.server.services.auth.authenticate(other.token)).actor,
).toBeUndefined();
expect(
(await env.server.services.auth.authenticate(otherPage)).actor,
).toBeUndefined();
// Asked from the sign-in: the page under it stays too.
await env.server.services.auth.revokeAllSessions(
(await actorOf(root.token))!,
);
expect(await actorOf(page)).toBeDefined();
});
it('is not counted as a sign-in', async () => {
const root = await signIn();
const before = await env.server.stores.session.listSignIns(userId, {
limit: 100,
});
await mint(root.token);
const after = await env.server.stores.session.listSignIns(userId, {
limit: 100,
});
expect(after.length).toBe(before.length);
});
});
@@ -1,165 +0,0 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { HttpError } from '../../core/http/HttpError.js';
import { PuterController } from '../types.js';
const ALLOWED_LAYOUTS = ['icons', 'details', 'list'];
const ALLOWED_SORT_BY = ['name', 'size', 'modified', 'type'];
const ALLOWED_SORT_ORDER = ['asc', 'desc'];
/**
* Folder view preference routes: layout and sort_by/sort_order, stored on the
* folder's fsentries row.
*/
/**
* Desktop preference writes — layout, sort order. Each persists on every call,
* and the GUI fires them on direct user action, so a per-minute ceiling well
* above human speed is enough to catch a stuck client.
*/
const PREFERENCE_WRITE_LIMIT = {
scope: 'desktop-preference',
limit: 120,
window: 60_000,
key: 'user',
};
export class DesktopController extends PuterController {
registerRoutes(router) {
// -- Folder layout -------------------------------------------
router.post(
'/set_layout',
{
subdomain: 'api',
requireUserActor: true,
allowFullAccessToken: true,
rateLimit: PREFERENCE_WRITE_LIMIT,
},
async (req, res) => {
const { item_uid, item_path, layout } = req.body ?? {};
if (!layout || !ALLOWED_LAYOUTS.includes(layout)) {
throw new HttpError(
400,
`\`layout\` must be one of: ${ALLOWED_LAYOUTS.join(', ')}`,
{ legacyCode: 'bad_request' },
);
}
await this.#updateFSEntry(
req.actor,
{ item_uid, item_path },
{ layout },
);
res.json({});
},
);
// -- Folder sort ---------------------------------------------
router.post(
'/set_sort_by',
{
subdomain: 'api',
requireUserActor: true,
allowFullAccessToken: true,
rateLimit: PREFERENCE_WRITE_LIMIT,
},
async (req, res) => {
const { item_uid, item_path, sort_by, sort_order } =
req.body ?? {};
if (!sort_by || !ALLOWED_SORT_BY.includes(sort_by)) {
throw new HttpError(
400,
`\`sort_by\` must be one of: ${ALLOWED_SORT_BY.join(', ')}`,
{ legacyCode: 'bad_request' },
);
}
const resolvedOrder = sort_order ?? 'asc';
if (!ALLOWED_SORT_ORDER.includes(resolvedOrder)) {
throw new HttpError(
400,
`\`sort_order\` must be one of: ${ALLOWED_SORT_ORDER.join(', ')}`,
{ legacyCode: 'bad_request' },
);
}
await this.#updateFSEntry(
req.actor,
{ item_uid, item_path },
{
sort_by,
sort_order: resolvedOrder,
},
);
res.json({});
},
);
}
// -- Helpers ------------------------------------------------------
/**
* Update columns on an actor-owned fsentry. Accepts either `item_uid` (fast
* path) or `item_path` (path lookups in FSEntryStore now have a
* recursive-CTE fallback for legacy rows with a NULL `path` column, so this
* works for old accounts too).
*
* Ownership: `entry.user_id === actor.user.id`. Kept as added validation
* against bad paths — the previous "drop user_id entirely" theory turned
* out to be wrong (the actual legacy issue was NULL paths, not user_id
* drift), so this filter doesn't lock out old accounts in practice.
*/
async #updateFSEntry(actor, { item_uid, item_path }, patch) {
if (!item_uid && !item_path) {
throw new HttpError(400, 'Missing `item_uid` or `item_path`', {
legacyCode: 'bad_request',
});
}
const entry = item_uid
? await this.stores.fsEntry.getEntryByUuid(item_uid)
: await this.stores.fsEntry.getEntryByPath(item_path);
if (!entry) {
throw new HttpError(404, 'Item not found', {
legacyCode: 'not_found',
});
}
const actorUserId = actor?.user?.id;
if (typeof actorUserId !== 'number' || entry.userId !== actorUserId) {
throw new HttpError(403, 'Not allowed to update this item', {
legacyCode: 'forbidden',
});
}
const keys = Object.keys(patch);
const setClause = keys.map((k) => `\`${k}\` = ?`).join(', ');
const values = keys.map((k) => patch[k]);
await this.clients.db.write(
`UPDATE \`fsentries\` SET ${setClause} WHERE \`id\` = ?`,
[...values, entry.id],
);
await this.stores.fsEntry.invalidateEntryCacheByUuid(entry.uuid);
}
onServerStart() {}
onServerPrepareShutdown() {}
onServerShutdown() {}
}
@@ -1,381 +0,0 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import type { Request, RequestHandler, Response } from 'express';
import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest';
import { v4 as uuidv4 } from 'uuid';
import type { Actor } from '../../core/actor.js';
import { PuterRouter } from '../../core/http/PuterRouter.js';
import { PuterServer } from '../../server.js';
import { setupTestServer } from '../../testUtil.js';
import { generateDefaultFsentries } from '../../util/userProvisioning.js';
// ── Test harness ────────────────────────────────────────────────────
//
// Boots one real PuterServer (in-memory sqlite + dynamo + s3 + mock
// redis) and re-registers DesktopController's inline lambda routes
// onto a fresh PuterRouter so each handler is reachable. Each test
// makes its own user via `makeUser` and exercises the live controller
// against the real wired stores (user, fsEntry) and DB client.
let server: PuterServer;
let router: PuterRouter;
beforeAll(async () => {
server = await setupTestServer();
router = new PuterRouter();
server.controllers.desktop.registerRoutes(router);
});
afterAll(async () => {
await server?.shutdown();
});
const makeUser = async (): Promise<{ actor: Actor; userId: number }> => {
const username = `dc-${Math.random().toString(36).slice(2, 10)}`;
const created = await server.stores.user.create({
username,
uuid: uuidv4(),
password: null,
email: `${username}@test.local`,
free_storage: 100 * 1024 * 1024,
requires_email_confirmation: false,
});
await generateDefaultFsentries(
server.clients.db,
server.stores.user,
created,
);
const refreshed = (await server.stores.user.getById(created.id))!;
return {
userId: refreshed.id,
actor: {
user: {
id: refreshed.id,
uuid: refreshed.uuid,
username: refreshed.username,
email: refreshed.email ?? null,
email_confirmed: true,
} as Actor['user'],
},
};
};
interface CapturedResponse {
statusCode: number;
body: unknown;
}
const makeReq = (init: {
body?: unknown;
actor?: Actor;
}): Request => {
return {
body: init.body ?? {},
query: {},
headers: {},
actor: init.actor,
} as unknown as Request;
};
const makeRes = () => {
const captured: CapturedResponse = { statusCode: 200, body: undefined };
const res = {
json: vi.fn((value: unknown) => {
captured.body = value;
return res;
}),
status: vi.fn((code: number) => {
captured.statusCode = code;
return res;
}),
setHeader: vi.fn(() => res),
};
return { res: res as unknown as Response, captured };
};
const findHandler = (method: string, path: string): RequestHandler => {
const route = router.routes.find(
(r) => r.method === method && r.path === path,
);
if (!route) throw new Error(`No ${method.toUpperCase()} ${path} route`);
return route.handler;
};
const callRoute = async (
method: string,
path: string,
req: Request,
res: Response,
) => {
const handler = findHandler(method, path);
await handler(req, res, () => {
throw new Error('handler called next() unexpectedly');
});
};
describe('DesktopController routes', () => {
it('has no desktop background route', () => {
expect(
router.routes.some((r) => r.path === '/set-desktop-bg'),
).toBe(false);
});
it('has no taskbar items route', () => {
expect(
router.routes.some((r) => r.path === '/update-taskbar-items'),
).toBe(false);
});
});
// ── /set_layout ─────────────────────────────────────────────────────
describe('DesktopController POST /set_layout', () => {
it('persists the new layout on the matching fsentry', async () => {
const { actor } = await makeUser();
const username = actor.user!.username!;
const documents = await server.stores.fsEntry.getEntryByPath(
`/${username}/Documents`,
);
expect(documents).not.toBeNull();
const { res, captured } = makeRes();
await callRoute(
'post',
'/set_layout',
makeReq({
body: { item_uid: documents!.uuid, layout: 'icons' },
actor,
}),
res,
);
expect(captured.body).toEqual({});
const refreshed = await server.stores.fsEntry.getEntryByUuid(
documents!.uuid,
);
expect(refreshed?.layout).toBe('icons');
});
it('resolves by item_path when item_uid is omitted', async () => {
const { actor } = await makeUser();
const username = actor.user!.username!;
const path = `/${username}/Pictures`;
const { res } = makeRes();
await callRoute(
'post',
'/set_layout',
makeReq({
body: { item_path: path, layout: 'list' },
actor,
}),
res,
);
const refreshed = await server.stores.fsEntry.getEntryByPath(path);
expect(refreshed?.layout).toBe('list');
});
it('throws 400 for an unknown layout value', async () => {
const { actor } = await makeUser();
const username = actor.user!.username!;
const documents = await server.stores.fsEntry.getEntryByPath(
`/${username}/Documents`,
);
const { res } = makeRes();
await expect(
callRoute(
'post',
'/set_layout',
makeReq({
body: { item_uid: documents!.uuid, layout: 'gallery' },
actor,
}),
res,
),
).rejects.toMatchObject({ statusCode: 400 });
});
it('throws 400 when neither item_uid nor item_path is supplied', async () => {
const { actor } = await makeUser();
const { res } = makeRes();
await expect(
callRoute(
'post',
'/set_layout',
makeReq({ body: { layout: 'icons' }, actor }),
res,
),
).rejects.toMatchObject({ statusCode: 400 });
});
it('throws 404 when the fsentry cannot be resolved', async () => {
const { actor } = await makeUser();
const { res } = makeRes();
await expect(
callRoute(
'post',
'/set_layout',
makeReq({
body: {
item_uid: '00000000-0000-0000-0000-000000000000',
layout: 'icons',
},
actor,
}),
res,
),
).rejects.toMatchObject({ statusCode: 404 });
});
it("throws 403 when the fsentry belongs to another user", async () => {
const owner = await makeUser();
const intruder = await makeUser();
const ownerUsername = owner.actor.user!.username!;
const ownerEntry = await server.stores.fsEntry.getEntryByPath(
`/${ownerUsername}/Documents`,
);
const { res } = makeRes();
await expect(
callRoute(
'post',
'/set_layout',
makeReq({
body: { item_uid: ownerEntry!.uuid, layout: 'icons' },
actor: intruder.actor,
}),
res,
),
).rejects.toMatchObject({ statusCode: 403 });
// Owner's entry remains untouched.
const refreshed = await server.stores.fsEntry.getEntryByUuid(
ownerEntry!.uuid,
);
expect(refreshed?.layout).toBeFalsy();
});
});
// ── /set_sort_by ────────────────────────────────────────────────────
describe('DesktopController POST /set_sort_by', () => {
it('persists sort_by + sort_order on the matching fsentry', async () => {
const { actor } = await makeUser();
const username = actor.user!.username!;
const documents = await server.stores.fsEntry.getEntryByPath(
`/${username}/Documents`,
);
const { res, captured } = makeRes();
await callRoute(
'post',
'/set_sort_by',
makeReq({
body: {
item_uid: documents!.uuid,
sort_by: 'name',
sort_order: 'desc',
},
actor,
}),
res,
);
expect(captured.body).toEqual({});
const refreshed = await server.stores.fsEntry.getEntryByUuid(
documents!.uuid,
);
expect(refreshed?.sortBy).toBe('name');
expect(refreshed?.sortOrder).toBe('desc');
});
it('defaults sort_order to "asc" when omitted', async () => {
const { actor } = await makeUser();
const username = actor.user!.username!;
const pictures = await server.stores.fsEntry.getEntryByPath(
`/${username}/Pictures`,
);
const { res } = makeRes();
await callRoute(
'post',
'/set_sort_by',
makeReq({
body: { item_uid: pictures!.uuid, sort_by: 'modified' },
actor,
}),
res,
);
const refreshed = await server.stores.fsEntry.getEntryByUuid(
pictures!.uuid,
);
expect(refreshed?.sortBy).toBe('modified');
expect(refreshed?.sortOrder).toBe('asc');
});
it('throws 400 for an unknown sort_by value', async () => {
const { actor } = await makeUser();
const username = actor.user!.username!;
const docs = await server.stores.fsEntry.getEntryByPath(
`/${username}/Documents`,
);
const { res } = makeRes();
await expect(
callRoute(
'post',
'/set_sort_by',
makeReq({
body: { item_uid: docs!.uuid, sort_by: 'random' },
actor,
}),
res,
),
).rejects.toMatchObject({ statusCode: 400 });
});
it('throws 400 for an invalid sort_order', async () => {
const { actor } = await makeUser();
const username = actor.user!.username!;
const docs = await server.stores.fsEntry.getEntryByPath(
`/${username}/Documents`,
);
const { res } = makeRes();
await expect(
callRoute(
'post',
'/set_sort_by',
makeReq({
body: {
item_uid: docs!.uuid,
sort_by: 'name',
sort_order: 'sideways',
},
actor,
}),
res,
),
).rejects.toMatchObject({ statusCode: 400 });
});
});
-2
View File
@@ -21,7 +21,6 @@ import { AppController } from './apps/AppController.js';
import { AppFeedbackController } from './feedback/AppFeedbackController.js';
import { AuthController } from './auth/AuthController.js';
import { BroadcastController } from './broadcast/BroadcastController.js';
import { DesktopController } from './desktop/DesktopController.js';
import { DriverController } from './drivers/DriverController.js';
import { EventsController } from './events/EventsController.js';
import { FSController } from './fs/FSController.js';
@@ -49,7 +48,6 @@ export const puterControllers = {
profile: ProfileController,
apps: AppController,
appFeedback: AppFeedbackController,
desktop: DesktopController,
hosting: HostingController,
system: SystemController,
fs: FSController,
@@ -44,7 +44,7 @@ import {
} from 'vitest';
import type { Actor } from '../../core/actor.js';
import type { ChatCompletionDriver } from '../../drivers/ai-chat/ChatCompletionDriver.js';
import type { ChatCompletionDriver } from '../../drivers/ai/interfaces/chat/ChatCompletionDriver.js';
import { PuterServer } from '../../server.js';
import { inRequestScope, setupTestServer } from '../../testUtil.js';
import { PuterAIController } from './PuterAIController.js';
@@ -49,8 +49,8 @@ import type { RouteOptions } from '../../core/http/index.js';
import {
COUNT_TOKENS,
type ChatCompletionDriver,
} from '../../drivers/ai-chat/ChatCompletionDriver.js';
import { AI_CONCURRENT, AI_RATE_LIMIT } from '../../drivers/util/aiLimits.js';
} from '../../drivers/ai/interfaces/chat/ChatCompletionDriver.js';
import { AI_CONCURRENT, AI_RATE_LIMIT } from '../../drivers/ai/shared/aiLimits.js';
import { PuterServer } from '../../server.js';
import { inRequestScope, setupTestServer } from '../../testUtil.js';
import { PuterAIController } from './PuterAIController.js';
@@ -30,25 +30,28 @@ import type { PuterRouter } from '../../core/http/PuterRouter.js';
import {
COUNT_TOKENS,
type ChatCompletionDriver,
} from '../../drivers/ai-chat/ChatCompletionDriver.js';
} from '../../drivers/ai/interfaces/chat/ChatCompletionDriver.js';
import type {
IChatCompleteResult,
IChatMessageResult,
IChatModel,
ICompleteArguments,
UsageDetails,
} from '../../drivers/ai-chat/types.js';
} from '../../drivers/ai/interfaces/chat/types.js';
import {
outputFormatFromResponseFormat,
rejectStatefulResponsesFields,
toolChoiceFromWire,
} from '../../drivers/ai-chat/utils/openaiParams.js';
} from '../../drivers/ai/protocols/openaiCompatible/openaiParams.js';
import {
promoteStopForToolCalls,
toFinishReason,
} from '../../drivers/ai-chat/utils/stopReason.js';
} from '../../drivers/ai/interfaces/chat/utils/stopReason.js';
import { isDriverStreamResult } from '../../drivers/meta.js';
import { AI_CONCURRENT, AI_RATE_LIMIT } from '../../drivers/util/aiLimits.js';
import {
AI_CONCURRENT,
AI_RATE_LIMIT,
} from '../../drivers/ai/shared/aiLimits.js';
import { PuterController } from '../types.js';
import { parseAnthropicRequest, toAnthropicMessage } from './anthropicWire.js';
import { pipeNdjsonStream } from './ndjsonStream.js';
@@ -18,7 +18,7 @@
*/
import { describe, expect, it } from 'vitest';
import type { IChatMessageResult } from '../../drivers/ai-chat/types.js';
import type { IChatMessageResult } from '../../drivers/ai/interfaces/chat/types.js';
import {
anthropicUsage,
parseAnthropicRequest,
@@ -32,7 +32,7 @@
import type { IncomingHttpHeaders } from 'node:http';
import { HttpError } from '../../core/http/HttpError.js';
import { sanitizeCacheControl } from '../../drivers/ai-chat/providers/claude/anthropicPolicy.js';
import { sanitizeCacheControl } from '../../drivers/ai/providers/claude/chat/anthropicPolicy.js';
import type {
ICompleteArguments,
IChatMessageResult,
@@ -42,9 +42,9 @@ import type {
ThinkingConfig,
ToolChoice,
UsageDetails,
} from '../../drivers/ai-chat/types.js';
import { needsOpenAICoercion } from '../../drivers/ai-chat/utils/normalizeToOpenAI.js';
import { fromFinishReason } from '../../drivers/ai-chat/utils/stopReason.js';
} from '../../drivers/ai/interfaces/chat/types.js';
import { needsOpenAICoercion } from '../../drivers/ai/interfaces/chat/utils/normalizeToOpenAI.js';
import { fromFinishReason } from '../../drivers/ai/interfaces/chat/utils/stopReason.js';
const toStringOrEmpty = (v: unknown): string =>
typeof v === 'string' ? v : '';
+4 -4
View File
@@ -20,15 +20,15 @@
/**
* SSE transport (`startSse`) plus the Anthropic `/v1/messages` streaming writer
* (`AnthropicSseWriter`), which turns the driver's NDJSON chunk protocol
* (`drivers/ai-chat/utils/Streaming.js`) into the exact Anthropic event
* sequence: `message_start` → `content_block_start/delta/stop`* →
* (`drivers/ai/interfaces/chat/utils/Streaming.js`) into the exact Anthropic
* event sequence: `message_start` → `content_block_start/delta/stop`* →
* `message_delta` → `message_stop`.
*/
import type { Response } from 'express';
import { onResponseClose } from '../../core/http/onResponseClose.js';
import type { UsageDetails } from '../../drivers/ai-chat/types.js';
import { promoteStopForToolCalls } from '../../drivers/ai-chat/utils/stopReason.js';
import type { UsageDetails } from '../../drivers/ai/interfaces/chat/types.js';
import { promoteStopForToolCalls } from '../../drivers/ai/interfaces/chat/utils/stopReason.js';
import { anthropicUsage } from './anthropicWire.js';
// -- SSE transport ------------------------------------------------------------
+7 -1
View File
@@ -76,8 +76,14 @@ export interface Actor {
/**
* Set when authenticated by a session token or an app-under-user token
* carrying one. `kind` mirrors the session row (`web`, `app`, `worker`).
* `parentUid` is the sign-in a `web` session was handed out under, set only
* on a first-party page's session.
*/
session?: { uid: string; kind?: string | null } | null;
session?: {
uid: string;
kind?: string | null;
parentUid?: string | null;
} | null;
/**
* Events handler runs behind this actor's writes; set only from an events
* handler token or a token one minted. Provenance, not authority, so
@@ -26,7 +26,7 @@ import type { IConfig } from '../../../types';
* Browsers send none of those variations, but config values do — a hand-edited
* `origin` with a trailing slash would otherwise reject every real request.
*/
const normalizeOrigin = (raw: string | undefined): string =>
export const normalizeOrigin = (raw: string | undefined): string =>
(raw ?? '').trim().replace(/\/+$/, '').toLowerCase();
/**
@@ -1,132 +0,0 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { describe, expect, it } from 'vitest';
import {
messagesHaveCompaction,
toAnthropicContextManagement,
toOpenAiContextManagement,
wantsCompaction,
} from './compaction.js';
describe('toOpenAiContextManagement', () => {
it('returns undefined when compaction is off', () => {
expect(toOpenAiContextManagement({})).toBeUndefined();
expect(
toOpenAiContextManagement({ compaction: false }),
).toBeUndefined();
});
it('builds a compaction entry from `true`', () => {
expect(toOpenAiContextManagement({ compaction: true })).toEqual([
{ type: 'compaction' },
]);
});
it('maps trigger_tokens to compact_threshold', () => {
expect(
toOpenAiContextManagement({ compaction: { trigger_tokens: 1000 } }),
).toEqual([{ type: 'compaction', compact_threshold: 1000 }]);
});
it('passes a raw context_management payload through verbatim', () => {
const raw = [{ type: 'compaction', compact_threshold: 5 }];
expect(
toOpenAiContextManagement({
compaction: true,
context_management: raw,
}),
).toBe(raw);
});
});
describe('toAnthropicContextManagement', () => {
it('returns undefined when compaction is off', () => {
expect(toAnthropicContextManagement({})).toBeUndefined();
});
it('builds a compact_20260112 edit from `true`', () => {
expect(toAnthropicContextManagement({ compaction: true })).toEqual({
edits: [{ type: 'compact_20260112' }],
});
});
it('maps trigger_tokens to an input_tokens trigger', () => {
expect(
toAnthropicContextManagement({
compaction: { trigger_tokens: 2000 },
}),
).toEqual({
edits: [
{
type: 'compact_20260112',
trigger: { type: 'input_tokens', value: 2000 },
},
],
});
});
it('passes a raw context_management payload through verbatim', () => {
const raw = { edits: [{ type: 'compact_20260112' }] };
expect(
toAnthropicContextManagement({ context_management: raw }),
).toBe(raw);
});
});
describe('wantsCompaction', () => {
it('is false without opt-in', () => {
expect(wantsCompaction({})).toBe(false);
expect(wantsCompaction({ compaction: false })).toBe(false);
});
it('is true for the neutral opt-in or a raw passthrough', () => {
expect(wantsCompaction({ compaction: true })).toBe(true);
expect(wantsCompaction({ compaction: { trigger_tokens: 1 } })).toBe(
true,
);
expect(wantsCompaction({ context_management: [] })).toBe(true);
});
});
describe('messagesHaveCompaction', () => {
it('detects a round-tripped compaction content block', () => {
expect(
messagesHaveCompaction([
{ role: 'user', content: [{ type: 'text', text: 'hi' }] },
{
role: 'assistant',
content: [
{ type: 'compaction', encrypted_content: 'ENC' },
],
},
]),
).toBe(true);
});
it('is false for ordinary messages or non-arrays', () => {
expect(
messagesHaveCompaction([
{ role: 'user', content: [{ type: 'text', text: 'hi' }] },
]),
).toBe(false);
expect(messagesHaveCompaction(undefined)).toBe(false);
expect(messagesHaveCompaction('nope')).toBe(false);
});
});
-718
View File
@@ -1,718 +0,0 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import {
DetectDocumentTextCommand,
InvalidS3ObjectException,
TextractClient,
UnsupportedDocumentException,
} from '@aws-sdk/client-textract';
import { Mistral } from '@mistralai/mistralai';
import { Actor } from '../../core/actor.js';
import { Context } from '../../core/context.js';
import { HttpError } from '../../core/http/HttpError.js';
import {
awsCredentialsFrom,
type AwsCredentials,
} from '../../util/awsCredentials.js';
import { mimeFromName } from '../../util/fileSigning.js';
import {
assertActorHasCredits,
insufficientCreditsError,
} from '../../services/metering/enforcement.js';
import type { CreditHold } from '../../services/metering/types.js';
import { PuterDriver } from '../types.js';
import {
type AiMeteringService,
withAiCostFactor,
} from '../util/aiCostFactor.js';
import { AI_CONCURRENT, AI_RATE_LIMIT } from '../util/aiLimits.js';
import {
type LoadedFile,
type ResolvedFileInput,
resolveFileInput,
} from '../util/fileInput.js';
import {
type ProviderCatalog,
ProviderRegistry,
readProviderKey,
} from '../util/providerRegistry.js';
import { OCR_COSTS } from './costs.js';
import {
DEFAULT_OCR_MODEL,
findOcrModel,
OCR_MAX_INPUT_BYTES,
OCR_MAX_PAGES,
RETIRED_OCR_MODELS,
type OcrModel,
type OcrProviderId,
} from './models.js';
import { countPdfPages } from './pdfPages.js';
import { textractRates } from './textractPricing.js';
/**
* Driver implementing `puter-ocr` — document OCR. Two providers: •
* `aws-textract` — AWS Textract (region-aware clients; direct S3 source when
* available) • `mistral` — Mistral OCR (URL/data-URL based)
*/
interface RecognizeArgs {
source?: unknown;
file?: unknown;
provider?: string;
model?: string;
// Mistral-specific options — ignored by Textract.
pages?: number[];
includeImageBase64?: boolean;
imageLimit?: number;
imageMinSize?: number;
bboxAnnotationFormat?: unknown;
documentAnnotationFormat?: unknown;
documentAnnotationPrompt?: string;
tableFormat?: string;
extractHeader?: boolean;
extractFooter?: boolean;
test_mode?: boolean;
}
/** One recognized line, in the same shape for every provider. */
interface OcrBlock {
type: string;
text: string;
confidence?: number;
page: number;
}
interface TextractBlock {
BlockType?: string;
Confidence?: number;
Text?: string;
}
interface MistralOcrResponse {
model?: string;
pages?: Array<{
index?: number;
markdown?: string;
images?: unknown[];
dimensions?: unknown;
}>;
documentAnnotation?: string | null;
usageInfo?: { pagesProcessed?: number };
}
interface MistralOcrClient {
ocr: {
process: (
payload: Record<string, unknown>,
) => Promise<MistralOcrResponse>;
};
}
// Aliases callers may use in place of a canonical provider id. Resolved here
// rather than in the SDK so a new alias reaches every caller at once.
const OCR_CATALOG: ProviderCatalog = {
label: 'OCR',
ids: ['aws-textract', 'mistral'],
defaultId: 'aws-textract',
aliases: {
aws: 'aws-textract',
'aws-textract': 'aws-textract',
textract: 'aws-textract',
mistral: 'mistral',
'mistral-ocr': 'mistral',
},
};
/** Recognizes one input with one provider's model. */
type OcrRecognizer = (
input: ResolvedFileInput,
args: RecognizeArgs,
model: OcrModel,
actor: Actor,
) => Promise<unknown>;
const NOT_CONFIGURED: Record<OcrProviderId, string> = {
'aws-textract': 'AWS credentials not configured',
mistral: 'Mistral OCR not configured',
};
const TABLE_FORMATS = new Set(['markdown', 'html']);
const DEFAULT_TEXTRACT_REGION = 'us-west-2';
const TEXTRACT_PAGE_USAGE_TYPE = 'aws-textract:detect-document-text:page';
const badRequest = (message: string) =>
new HttpError(400, message, { legacyCode: 'bad_request' });
/**
* Accept a Mistral annotation format in the REST spelling
* (`json_schema.schema`) or the SDK spelling (`jsonSchema.schemaDefinition`)
* and return the SDK shape.
*/
const toMistralResponseFormat = (
name: string,
value: unknown,
): Record<string, unknown> => {
const format = value as Record<string, unknown> | null;
const jsonSchema = (format?.jsonSchema ?? format?.json_schema) as
Record<string, unknown> | undefined;
const schema = jsonSchema?.schemaDefinition ?? jsonSchema?.schema;
if (
!format ||
typeof format !== 'object' ||
(format.type !== undefined && format.type !== 'json_schema') ||
!schema ||
typeof schema !== 'object'
) {
throw badRequest(
`\`${name}\` must be { type: 'json_schema', json_schema: { name, schema } }`,
);
}
return {
type: 'json_schema',
jsonSchema: {
name:
typeof jsonSchema?.name === 'string'
? jsonSchema.name
: 'annotation',
schemaDefinition: schema,
...(typeof jsonSchema?.description === 'string' && {
description: jsonSchema.description,
}),
...(typeof jsonSchema?.strict === 'boolean' && {
strict: jsonSchema.strict,
}),
},
};
};
const inputMimeType = (loaded: LoadedFile): string =>
loaded.mimeType ??
mimeFromName(loaded.filename) ??
'application/octet-stream';
/**
* Declared documents (PDF, DOCX, PPTX, ...) and PDF bytes are documents; images
* and untyped bytes are read as a single image.
*/
const isDocumentInput = (loaded: LoadedFile): boolean => {
const mime = inputMimeType(loaded);
return (
loaded.buffer.subarray(0, 4).toString('latin1') === '%PDF' ||
loaded.filename.toLowerCase().endsWith('.pdf') ||
(!mime.startsWith('image/') && mime !== 'application/octet-stream')
);
};
/**
* The most pages a call can bill, known before the provider runs: a PDF's page
* count, or a size-based guess for other documents, capped by a `pages`
* selection and by what the provider reads in one call. Reads the input only
* for a provider that takes more than one page.
*/
const estimateOcrPages = async (
input: ResolvedFileInput,
provider: OcrProviderId,
selection?: unknown,
): Promise<number> => {
const maxPages = OCR_MAX_PAGES[provider];
if (maxPages <= 1) return 1;
const loaded = await input.load();
if (!isDocumentInput(loaded)) return 1;
// A document we can't count is billed in proportion to its size, reaching
// the provider's page limit at its size limit (20 pages per MB for
// Mistral). Small files stay cheap; a large one can't pass as one page.
const documentPages =
(await countPdfPages(loaded.buffer, maxPages)) ??
Math.ceil(
(loaded.buffer.length * maxPages) / OCR_MAX_INPUT_BYTES[provider],
);
const selected = Array.isArray(selection)
? new Set(
selection.filter(
(page) => Number.isInteger(page) && (page as number) >= 0,
),
).size
: 0;
return Math.max(1, Math.min(documentPages, maxPages, selected || Infinity));
};
export class OCRDriver extends PuterDriver {
readonly driverInterface = 'puter-ocr';
readonly driverName = 'ai-ocr';
// Shared AI policy — see `drivers/util/aiLimits.ts` for the tier table.
readonly rateLimit = AI_RATE_LIMIT;
readonly concurrent = AI_CONCURRENT;
/** Metering scoped to this driver. Lazy: services wire up after drivers. */
get #aiMetering(): AiMeteringService {
return withAiCostFactor(
this.services.metering,
this.clients.event,
this.driverName,
);
}
/**
* Textract is reported at the configured region's first-tier rate. A call
* that reads its file from a bucket in another region is billed at that
* region's rate instead, which this per-usage-type report can't show.
*/
override getReportedCosts() {
const textractRegion =
this.#awsConfig?.region ?? DEFAULT_TEXTRACT_REGION;
return Object.entries(OCR_COSTS).map(([usageType, ucentsPerUnit]) => ({
usageType,
ucentsPerUnit:
usageType === TEXTRACT_PAGE_USAGE_TYPE
? this.#textractRates.peek(textractRegion)
: ucentsPerUnit,
unit: 'page',
source: 'driver:aiOcr',
}));
}
// Older SDK bundles name the provider in the driver slot instead of
// passing `{ provider }`; `#resolveModel` reads the requested alias
// back off the Context.
readonly driverAliases = [...OCR_CATALOG.ids];
readonly isDefault = true;
// Textract state — one client per region.
#textractClients: Record<string, TextractClient> = {};
#awsConfig: { credentials: AwsCredentials; region: string } | null = null;
/** DetectDocumentText rates per region, from AWS's public price list. */
#textractRates = textractRates(OCR_COSTS[TEXTRACT_PAGE_USAGE_TYPE]);
// Mistral state.
#mistral: MistralOcrClient | null = null;
/** Configured providers, each a recognizer bound to its state above. */
#providers = new ProviderRegistry<OcrRecognizer>(OCR_CATALOG);
override onServerStart() {
const providers = this.config.providers ?? {};
const textract = providers['aws-textract'] as
Record<string, unknown> | undefined;
const textractAws = (textract?.aws ?? textract) as
Record<string, unknown> | undefined;
const textractCredentials = awsCredentialsFrom(textractAws);
const textractRegion =
(textractAws?.region as string | undefined) ??
(textract?.region as string | undefined) ??
DEFAULT_TEXTRACT_REGION;
if (textractCredentials) {
this.#awsConfig = {
credentials: textractCredentials,
region: textractRegion,
};
// Warm the configured region so the cost report has its rate.
void this.#textractRates.rate(textractRegion);
this.#providers.register(
'aws-textract',
(input, _args, model, actor) =>
this.#textractRecognize(input, model, actor),
);
}
const mistralKey = readProviderKey(providers['mistral-ocr']);
if (mistralKey) {
try {
this.#mistral = new Mistral({
apiKey: mistralKey,
}) as unknown as MistralOcrClient;
this.#providers.register(
'mistral',
(input, args, model, actor) =>
this.#mistralRecognize(input, args, model, actor),
);
} catch (e) {
console.warn(
'[OCRDriver] Failed to init Mistral:',
(e as Error).message,
);
}
}
}
async recognize(args: RecognizeArgs) {
if (args.test_mode) return sampleResponse();
const model = this.#resolveModel(args);
const actor = Context.get('actor');
if (!actor)
throw new HttpError(401, 'Authentication required', {
legacyCode: 'unauthorized',
});
const input = args.source ?? args.file;
if (!input) throw badRequest('`source` is required');
const recognize = this.#providers.get(model.provider);
if (!recognize)
throw new HttpError(500, NOT_CONFIGURED[model.provider], {
legacyCode: 'internal_error',
});
// Cheap, cached check: reject a zero-balance actor before paying for
// the file load and page estimate, which a real credit check would
// reject anyway once it runs.
await assertActorHasCredits(this.services.metering, actor, this.config);
// A Puter file is resolved and access-checked here but read only if
// the provider needs its bytes; Textract reads it straight from S3.
const resolved = await resolveFileInput(
this.stores,
this.services.fs,
actor,
input,
{
acceptWebInput: true,
maxBytes: OCR_MAX_INPUT_BYTES[model.provider],
},
);
return recognize(resolved, args, model, actor);
}
/**
* Decide which model handles a call. A `model` picks its own provider; an
* explicit `provider`, then the legacy driver alias the caller dispatched
* through, then whichever provider is configured pick that provider's
* default model.
*/
#resolveModel(args: RecognizeArgs): OcrModel {
const provider = args.provider
? (this.#providers.resolve(args.provider) as OcrProviderId)
: undefined;
if (args.model !== undefined) {
if (typeof args.model !== 'string' || !args.model.trim())
throw badRequest('`model` must be a non-empty string');
const retired = RETIRED_OCR_MODELS[args.model.trim().toLowerCase()];
if (retired)
throw badRequest(
`${args.model} is no longer available: ${retired}`,
);
const model = findOcrModel(args.model);
if (!model) throw badRequest(`Unknown OCR model: ${args.model}`);
if (provider && model.provider !== provider)
throw badRequest(
`OCR model ${args.model} is not served by provider ${args.provider}`,
);
return model;
}
const hinted =
provider ?? this.#providers.normalize(Context.get('driverName'));
if (!hinted && this.#providers.names().length === 0)
throw new HttpError(500, 'No OCR provider configured', {
legacyCode: 'internal_error',
});
const chosen = (hinted ?? this.#providers.defaultId()) as OcrProviderId;
return findOcrModel(DEFAULT_OCR_MODEL[chosen])!;
}
/**
* Refuse a call the balance can't cover, and hold its cost while the
* provider runs so the account's concurrent calls see it. Usage is still
* metered from the pages the provider reports.
*/
async #holdCredits(
actor: Actor,
usageType: string,
cost: number,
): Promise<CreditHold> {
// Priced at the cost factor usage is recorded at.
const hold = await this.#aiMetering.reserveAiCredits(
actor,
usageType,
cost,
);
if (!hold) throw insufficientCreditsError();
return hold;
}
// -- AWS Textract -------------------------------------------------
#textractClientFor(region: string): TextractClient {
const cached = this.#textractClients[region];
if (cached) return cached;
const client = new TextractClient({
credentials: this.#awsConfig!.credentials,
region,
});
this.#textractClients[region] = client;
return client;
}
async #textractRecognize(
input: ResolvedFileInput,
model: OcrModel,
actor: Actor,
) {
// Prefer S3 direct source if the file is FS-backed; fall back to raw bytes.
const { fsEntry } = input;
const s3Info =
fsEntry && fsEntry.bucket && fsEntry.bucketRegion
? {
bucket: fsEntry.bucket,
bucketRegion: fsEntry.bucketRegion,
key: fsEntry.uuid,
}
: null;
const configuredRegion =
this.#awsConfig!.region ?? DEFAULT_TEXTRACT_REGION;
// Both regions the call may run in are priced up front, so the hold
// covers either and a refresh mid-call can't change the charge.
const [configuredRate, s3Rate] = await Promise.all([
this.#textractRates.rate(configuredRegion),
s3Info ? this.#textractRates.rate(s3Info.bucketRegion) : undefined,
]);
const tryRun = async (useS3: boolean) => {
const region =
s3Info && useS3 ? s3Info.bucketRegion : configuredRegion;
const client = this.#textractClientFor(region);
const document =
s3Info && useS3
? { S3Object: { Bucket: s3Info.bucket, Name: s3Info.key } }
: { Bytes: (await input.load()).buffer };
return client.send(
new DetectDocumentTextCommand({ Document: document }),
);
};
const hold = await this.#holdCredits(
actor,
model.pageUsageType,
Math.max(configuredRate, s3Rate ?? 0) *
(await estimateOcrPages(input, model.provider)),
);
try {
let response;
let costPerPage: number;
try {
try {
response = await tryRun(Boolean(s3Info));
costPerPage = s3Rate ?? configuredRate;
} catch (err) {
if (!(s3Info && err instanceof InvalidS3ObjectException))
throw err;
response = await tryRun(false);
costPerPage = configuredRate;
}
} catch (err) {
if (err instanceof UnsupportedDocumentException)
throw badRequest(
'AWS Textract reads JPEG, PNG, TIFF and single-page PDF documents; use a Mistral OCR model for multi-page PDFs and other formats',
);
throw err;
}
const blocks: OcrBlock[] = [];
let pageCount = 0;
for (const block of (response.Blocks ?? []) as TextractBlock[]) {
if (block.BlockType === 'PAGE') {
pageCount += 1;
continue;
}
if (block.BlockType !== 'LINE' || !block.Text) continue;
blocks.push({
type: 'text/textract:LINE',
text: block.Text,
confidence: Number(block.Confidence ?? 0),
page: Math.max(pageCount - 1, 0),
});
}
const pages = pageCount || 1;
this.#aiMetering.incrementUsage(
actor,
model.pageUsageType,
pages,
costPerPage * pages,
);
return {
model: model.id,
blocks,
text: blocks.map((b) => b.text).join('\n'),
};
} finally {
await hold.release();
}
}
// -- Mistral OCR --------------------------------------------------
async #mistralRecognize(
input: ResolvedFileInput,
args: RecognizeArgs,
model: OcrModel,
actor: Actor,
) {
const loaded = await input.load();
const payload: Record<string, unknown> = {
model: model.id,
document: this.#mistralBuildChunk(loaded),
};
if (args.pages) payload.pages = args.pages;
if (args.includeImageBase64 !== undefined)
payload.includeImageBase64 = args.includeImageBase64;
if (typeof args.imageLimit === 'number')
payload.imageLimit = args.imageLimit;
if (typeof args.imageMinSize === 'number')
payload.imageMinSize = args.imageMinSize;
if (args.bboxAnnotationFormat !== undefined)
payload.bboxAnnotationFormat = toMistralResponseFormat(
'bboxAnnotationFormat',
args.bboxAnnotationFormat,
);
if (args.documentAnnotationFormat !== undefined)
payload.documentAnnotationFormat = toMistralResponseFormat(
'documentAnnotationFormat',
args.documentAnnotationFormat,
);
if (typeof args.documentAnnotationPrompt === 'string')
payload.documentAnnotationPrompt = args.documentAnnotationPrompt;
if (args.tableFormat !== undefined) {
if (!TABLE_FORMATS.has(args.tableFormat))
throw badRequest("`tableFormat` must be 'markdown' or 'html'");
payload.tableFormat = args.tableFormat;
}
if (typeof args.extractHeader === 'boolean')
payload.extractHeader = args.extractHeader;
if (typeof args.extractFooter === 'boolean')
payload.extractFooter = args.extractFooter;
const annotations =
payload.documentAnnotationFormat !== undefined ||
payload.bboxAnnotationFormat !== undefined;
const costPerPage =
OCR_COSTS[model.pageUsageType] +
(annotations && model.annotationUsageType
? OCR_COSTS[model.annotationUsageType]
: 0);
const hold = await this.#holdCredits(
actor,
model.pageUsageType,
costPerPage *
(await estimateOcrPages(input, model.provider, args.pages)),
);
try {
const response = await this.#mistral!.ocr.process(payload);
this.#recordMistralUsage(response, model, actor, annotations);
return this.#normalizeMistralResponse(response, model);
} finally {
await hold.release();
}
}
#mistralBuildChunk(loaded: LoadedFile): Record<string, unknown> {
const mime = inputMimeType(loaded);
const dataUrl = `data:${mime};base64,${loaded.buffer.toString('base64')}`;
if (isDocumentInput(loaded)) {
return {
type: 'document_url',
documentUrl: dataUrl,
documentName: loaded.filename,
};
}
return { type: 'image_url', imageUrl: { url: dataUrl } };
}
#normalizeMistralResponse(response: MistralOcrResponse, model: OcrModel) {
const pages = response?.pages ?? [];
const blocks: OcrBlock[] = [];
for (const [position, page] of pages.entries()) {
if (typeof page?.markdown !== 'string') continue;
const lines = page.markdown
.split('\n')
.map((l) => l.trim())
.filter(Boolean);
for (const line of lines) {
blocks.push({
type: 'text/mistral:LINE',
text: line,
page: page.index ?? position,
});
}
}
return {
model: response?.model ?? model.id,
pages,
usage_info: response?.usageInfo,
...(typeof response?.documentAnnotation === 'string' && {
document_annotation: response.documentAnnotation,
}),
blocks,
text: blocks.map((b) => b.text).join('\n'),
};
}
#recordMistralUsage(
response: MistralOcrResponse,
model: OcrModel,
actor: Actor,
annotations: boolean,
) {
try {
const pagesProcessed =
response?.usageInfo?.pagesProcessed ??
(Array.isArray(response?.pages) ? response.pages.length : 1);
this.#aiMetering.incrementUsage(
actor,
model.pageUsageType,
pagesProcessed,
OCR_COSTS[model.pageUsageType] * pagesProcessed,
);
if (annotations && model.annotationUsageType) {
this.#aiMetering.incrementUsage(
actor,
model.annotationUsageType,
pagesProcessed,
OCR_COSTS[model.annotationUsageType] * pagesProcessed,
);
}
} catch {
// Non-critical.
}
}
}
function sampleResponse() {
return {
model: 'test-mode',
blocks: [
{
type: 'text/puter:sample-output',
confidence: 1,
text: 'test_mode is enabled; this is a sample OCR response.',
page: 0,
},
],
text: 'test_mode is enabled; this is a sample OCR response.',
};
}
@@ -1,511 +0,0 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { pipeline, Readable, Transform } from 'node:stream';
import { Context } from '../../core/context.js';
import { HttpError } from '../../core/http/HttpError.js';
import type { DriverStreamResult } from '../meta.js';
import { OUTPUT_FORMATS } from '../ai-tts/providers/elevenlabs/ElevenLabsTTSProvider.js';
import { PuterDriver } from '../types.js';
import {
type AiMeteringService,
withAiCostFactor,
withStreamedAiCharge,
} from '../util/aiCostFactor.js';
import { AI_CONCURRENT, AI_RATE_LIMIT } from '../util/aiLimits.js';
import { loadFileInput } from '../util/fileInput.js';
import {
type ProviderCatalog,
ProviderRegistry,
readProviderKey,
} from '../util/providerRegistry.js';
import { SAMPLE_AUDIO_URL } from '../util/testMode.js';
import { upstreamBodyStream, upstreamFetch } from '../util/upstreamErrors.js';
import { VOICE_CHANGER_COSTS } from './costs.js';
/**
* Driver implementing `puter-speech2speech` — voice changer. Currently a single
* provider (ElevenLabs).
*/
const DEFAULT_MODEL = 'eleven_multilingual_sts_v2';
const DEFAULT_VOICE_ID = '21m00Tcm4TlvDq8ikWAM';
const DEFAULT_OUTPUT_FORMAT = 'mp3_44100_128';
const MAX_AUDIO_FILE_SIZE = 25 * 1024 * 1024;
// Covers the upload as well as the conversion.
const CONVERT_TIMEOUT_MS = 10 * 60 * 1000;
const SUPPORTED_MODELS = Object.keys(VOICE_CHANGER_COSTS).map(
(key) => key.split(':')[1]!,
);
const VOICE_CHANGER_CATALOG: ProviderCatalog = {
label: 'Speech-to-speech',
ids: ['elevenlabs'],
defaultId: 'elevenlabs',
aliases: { elevenlabs: 'elevenlabs' },
};
interface ElevenLabsConfig {
apiKey: string;
baseUrl: string;
defaultVoiceId: string;
defaultModelId: string;
}
interface ConvertArgs {
audio: unknown;
provider?: string;
voice?: string;
voice_id?: string;
voiceId?: string;
model?: string;
model_id?: string;
voice_settings?: unknown;
voiceSettings?: unknown;
seed?: number;
remove_background_noise?: boolean;
output_format?: string;
file_format?: string;
optimize_streaming_latency?: number;
enable_logging?: boolean;
test_mode?: boolean;
}
export class VoiceChangerDriver extends PuterDriver {
readonly driverInterface = 'puter-speech2speech';
readonly driverName = 'ai-speech2speech';
// Older SDK bundles name the provider in the driver slot.
readonly driverAliases = ['elevenlabs-voice-changer'];
readonly isDefault = true;
// Shared AI policy — see `drivers/util/aiLimits.ts` for the tier table.
readonly rateLimit = AI_RATE_LIMIT;
readonly concurrent = AI_CONCURRENT;
/** Metering scoped to this driver. Lazy: services wire up after drivers. */
get #aiMetering(): AiMeteringService {
return withAiCostFactor(
this.services.metering,
this.clients.event,
this.driverName,
);
}
override getReportedCosts(): Record<string, unknown>[] {
return Object.entries(VOICE_CHANGER_COSTS).map(
([usageType, ucentsPerUnit]) => ({
usageType,
ucentsPerUnit,
unit: 'second',
source: 'driver:aiSpeech2Speech',
}),
);
}
#providers = new ProviderRegistry<ElevenLabsConfig>(VOICE_CHANGER_CATALOG);
override onServerStart() {
const elevenlabs = this.config.providers?.elevenlabs;
const apiKey = readProviderKey(elevenlabs);
if (!apiKey) return;
this.#providers.register('elevenlabs', {
apiKey,
baseUrl:
(elevenlabs?.apiBaseUrl as string | undefined) ??
'https://api.elevenlabs.io',
defaultVoiceId:
(elevenlabs?.defaultVoiceId as string | undefined) ??
DEFAULT_VOICE_ID,
defaultModelId:
(elevenlabs?.speechToSpeechModelId as string | undefined) ??
DEFAULT_MODEL,
});
}
async convert(
args: ConvertArgs,
): Promise<DriverStreamResult | { url: string; content_type: string }> {
// Only one provider exists today, but naming a different one should
// fail loudly rather than quietly convert with this one.
const providerName = this.#providers.resolve(args.provider);
if (args.test_mode) {
return { url: SAMPLE_AUDIO_URL, content_type: 'audio/mpeg' };
}
const elevenlabs = this.#providers.get(providerName);
if (!elevenlabs) {
throw new HttpError(500, 'ElevenLabs API key not configured', {
legacyCode: 'internal_error',
});
}
const actor = Context.get('actor');
if (!actor)
throw new HttpError(401, 'Authentication required', {
legacyCode: 'unauthorized',
});
if (!args.audio) {
throw new HttpError(400, '`audio` is required', {
legacyCode: 'bad_request',
});
}
const modelId =
args.model_id || args.model || elevenlabs.defaultModelId;
const voiceId =
args.voice_id ||
args.voiceId ||
args.voice ||
elevenlabs.defaultVoiceId;
if (!voiceId)
throw new HttpError(400, '`voice` is required', {
legacyCode: 'bad_request',
});
// `voiceId` lands in the request URL path, which carries our API key,
// so anything outside ElevenLabs' `[A-Za-z0-9_-]+` id shape could
// steer the request at a different endpoint.
if (!/^[A-Za-z0-9_-]+$/.test(voiceId))
throw new HttpError(400, '`voice` must be alphanumeric', {
legacyCode: 'bad_request',
});
// Gate on the cost table: an id we can't price is one we can't bill
// for, and ElevenLabs bills us for it either way. This also keeps
// anything but a known id out of the multipart `model_id` field.
const usageKey = `elevenlabs:${modelId}:second`;
if (!Object.hasOwn(VOICE_CHANGER_COSTS, usageKey)) {
throw new HttpError(
400,
`Invalid model: ${modelId}. Expected: ${SUPPORTED_MODELS.join(', ')}`,
{
legacyCode: 'field_invalid',
fields: {
key: 'model',
expected: SUPPORTED_MODELS,
got: modelId,
},
},
);
}
const ucentsPerSecond = VOICE_CHANGER_COSTS[usageKey]!;
const loaded = await loadFileInput(
this.stores,
this.services.fs,
actor,
args.audio,
{ maxBytes: MAX_AUDIO_FILE_SIZE, acceptWebInput: true },
);
// The hold estimates seconds from the input at 128 kbps: compact audio
// is under-held rather than refused. Usage is metered from the length
// of the audio that comes back.
const estimatedSeconds = Math.max(
1,
Math.ceil(loaded.buffer.byteLength / 16000),
);
const estimatedCost = ucentsPerSecond * estimatedSeconds;
const formData = new FormData();
const blob = new Blob([loaded.buffer as BlobPart], {
type: loaded.mimeType ?? 'application/octet-stream',
});
formData.append('audio', blob, loaded.filename);
formData.append('model_id', modelId);
const settings = args.voice_settings ?? args.voiceSettings;
if (settings !== undefined && settings !== null) {
formData.append(
'voice_settings',
typeof settings === 'string'
? settings
: JSON.stringify(settings),
);
}
if (args.seed !== undefined && args.seed !== null) {
formData.append('seed', String(args.seed));
}
if (typeof args.remove_background_noise === 'boolean') {
formData.append(
'remove_background_noise',
String(args.remove_background_noise),
);
}
if (args.file_format) {
formData.append('file_format', args.file_format);
}
const searchParams = new URLSearchParams();
const outputFormat = args.output_format || DEFAULT_OUTPUT_FORMAT;
if (outputFormat) searchParams.set('output_format', outputFormat);
if (
args.optimize_streaming_latency !== undefined &&
args.optimize_streaming_latency !== null
) {
searchParams.set(
'optimize_streaming_latency',
String(args.optimize_streaming_latency),
);
}
if (args.enable_logging !== undefined && args.enable_logging !== null) {
searchParams.set('enable_logging', String(args.enable_logging));
}
const url = new URL(
`/v1/speech-to-speech/${voiceId}`,
elevenlabs.baseUrl,
);
const search = searchParams.toString();
if (search) url.search = search;
const metering = this.#aiMetering;
const output = measuredOutput(
outputBytesPerSecond(outputFormat),
outputFormat,
);
return withStreamedAiCharge(
metering,
actor,
usageKey,
estimatedCost,
async (): Promise<DriverStreamResult> => {
const response = await upstreamFetch(
'elevenlabs',
url,
{
method: 'POST',
headers: { 'xi-api-key': elevenlabs.apiKey },
body: formData,
},
{
providerLabel: 'ElevenLabs',
timeoutMs: CONVERT_TIMEOUT_MS,
stream: true,
},
);
return {
dataType: 'stream',
content_type:
response.headers.get('content-type') ?? 'audio/mpeg',
stream: output.measure(upstreamBodyStream(response)),
};
},
async () => {
const measured = await output.seconds;
// A stream cut short was still converted in full upstream.
const seconds =
measured.complete && measured.seconds
? measured.seconds
: Math.max(measured.seconds ?? 0, estimatedSeconds);
return metering.incrementUsage(
actor,
usageKey,
seconds,
ucentsPerSecond * seconds,
);
},
);
}
}
/**
* Bytes per second of an ElevenLabs `output_format`. A format outside the
* documented set is sized as the default, so a made-up bitrate can't shrink the
* bill.
*/
export function outputBytesPerSecond(format: string): number {
const known = OUTPUT_FORMATS.has(format) ? format : DEFAULT_OUTPUT_FORMAT;
const [codec, rate, kbps] = known.split('_');
if (codec === 'mp3' || codec === 'opus') return Number(kbps) * 125;
if (codec === 'ulaw' || codec === 'alaw') return Number(rate);
// pcm and wav: 16-bit mono.
return Number(rate) * 2;
}
// Past this, a WAV whose data chunk hasn't started is billed by raw length.
const WAV_HEADER_MAX_BYTES = 64 * 1024;
/**
* Where the samples of a RIFF/WAVE stream start, once `head` reaches its `data`
* chunk; undefined until then, or if `head` isn't a WAV.
*/
export function wavDataOffset(head: Buffer): number | undefined {
if (head.length < 12) return undefined;
if (
head.toString('ascii', 0, 4) !== 'RIFF' ||
head.toString('ascii', 8, 12) !== 'WAVE'
)
return undefined;
let offset = 12;
while (offset + 8 <= head.length) {
if (head.toString('ascii', offset, offset + 4) === 'data')
return offset + 8;
const size = head.readUInt32LE(offset + 4);
offset += 8 + size + (size & 1);
}
return undefined;
}
const OGG_HEADER_BYTES = 27;
const NO_GRANULE = 2n ** 64n - 1n;
/**
* Walks an Ogg Opus stream page by page as it arrives. Opus is variable
* bitrate, so its length comes from the last page's granule position (48 kHz
* samples, less the stream's pre-skip) rather than its byte count.
*/
export class OggOpusClock {
#header = Buffer.alloc(0);
#skip = 0;
#firstBody: Buffer | undefined = Buffer.alloc(0);
#firstBodyBytes = 0;
#granule: bigint | undefined;
#preSkip = 0;
#broken = false;
push(chunk: Buffer): void {
let offset = 0;
while (!this.#broken && offset < chunk.length) {
if (this.#skip > 0) {
const n = Math.min(this.#skip, chunk.length - offset);
if (this.#firstBody && this.#firstBody.length < 19) {
this.#firstBody = Buffer.concat([
this.#firstBody,
chunk.subarray(offset, offset + n),
]);
}
this.#skip -= n;
offset += n;
if (this.#skip === 0 && this.#firstBody) this.#readOpusHead();
continue;
}
const want =
this.#header.length < OGG_HEADER_BYTES
? OGG_HEADER_BYTES
: OGG_HEADER_BYTES + this.#header[26]!;
const n = Math.min(
want - this.#header.length,
chunk.length - offset,
);
this.#header = Buffer.concat([
this.#header,
chunk.subarray(offset, offset + n),
]);
offset += n;
if (this.#header.length < want) continue;
if (this.#header.toString('ascii', 0, 4) !== 'OggS') {
this.#broken = true;
return;
}
if (want === OGG_HEADER_BYTES && this.#header[26]! > 0) continue;
const granule = this.#header.readBigUInt64LE(6);
if (granule !== NO_GRANULE) this.#granule = granule;
let body = 0;
for (const lacing of this.#header.subarray(OGG_HEADER_BYTES)) {
body += lacing;
}
this.#header = Buffer.alloc(0);
this.#skip = body;
if (this.#firstBody) this.#firstBodyBytes = body;
}
}
/** Whole seconds so far, once the stream has proven to be Ogg Opus. */
get seconds(): number | undefined {
if (this.#broken || this.#granule === undefined) return undefined;
const samples = Number(this.#granule) - this.#preSkip;
return samples > 0
? Math.max(1, Math.ceil(samples / 48_000))
: undefined;
}
#readOpusHead(): void {
const head = this.#firstBody!;
this.#firstBody = undefined;
if (
this.#firstBodyBytes < 19 ||
head.toString('ascii', 0, 8) !== 'OpusHead'
) {
this.#broken = true;
return;
}
this.#preSkip = head.readUInt16LE(10);
}
}
/**
* Counts the bytes of a converted stream. `seconds` settles when the stream
* ends or closes: its length in whole seconds (undefined when nothing came
* back), and whether the upstream sent all of it. A WAV's header isn't audio,
* so it isn't counted; Opus is timed by its pages.
*/
function measuredOutput(bytesPerSecond: number, format: string) {
let bytes = 0;
let head: Buffer | undefined = format.startsWith('wav_')
? Buffer.alloc(0)
: undefined;
const opus = format.startsWith('opus_') ? new OggOpusClock() : undefined;
let headerBytes = 0;
let settle!: (measured: { seconds?: number; complete: boolean }) => void;
const seconds = new Promise<{ seconds?: number; complete: boolean }>(
(resolve) => {
settle = resolve;
},
);
const lengthOf = () => {
const timed = opus?.seconds;
if (timed !== undefined) return timed;
const audio = bytes - headerBytes;
return audio > 0
? Math.max(1, Math.ceil(audio / bytesPerSecond))
: undefined;
};
const measure = (source: Readable): Readable => {
const counter = new Transform({
transform(chunk: Buffer, _encoding, callback) {
bytes += chunk.length;
opus?.push(chunk);
if (head) {
head = Buffer.concat([head, chunk]);
const offset = wavDataOffset(head);
if (offset !== undefined) headerBytes = offset;
if (
offset !== undefined ||
head.length >= WAV_HEADER_MAX_BYTES
)
head = undefined;
}
callback(null, chunk);
},
flush(callback) {
settle({ seconds: lengthOf(), complete: true });
callback();
},
});
counter.once('close', () =>
settle({ seconds: lengthOf(), complete: false }),
);
pipeline(source, counter, () => {});
return counter;
};
return { measure, seconds };
}
@@ -19,9 +19,9 @@
import { createServer, type Server } from 'node:http';
import { afterAll, beforeAll, beforeEach, describe, expect, it } from 'vitest';
import { setupPuterTestEnv, type PuterTestEnv } from '../../testUtil.js';
import { setupPuterTestEnv, type PuterTestEnv } from '../../../../testUtil.js';
import { clearUnhealthyRoutes } from './utils/providerHealth.js';
import { makeActor } from '../../core/actor.js';
import { makeActor } from '../../../../core/actor.js';
const policyMessage =
'Policy Violation: this user has been blocked for a previous policy violation. Learn more: https://vendor.test/private';
@@ -19,8 +19,8 @@
import { createServer, type Server, type ServerResponse } from 'node:http';
import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest';
import { makeActor, type Actor } from '../../core/actor.js';
import { setupPuterTestEnv, type PuterTestEnv } from '../../testUtil.js';
import { makeActor, type Actor } from '../../../../core/actor.js';
import { setupPuterTestEnv, type PuterTestEnv } from '../../../../testUtil.js';
const routes = [
'/drivers/call',
@@ -42,22 +42,22 @@ import {
vi,
} from 'vitest';
import { Context } from '../../core/context.js';
import { HttpError } from '../../core/http/HttpError.js';
import { PuterServer } from '../../server.js';
import { setupTestServer } from '../../testUtil.js';
import { withTestActor } from '../integrationTestUtil.js';
import { Context } from '../../../../core/context.js';
import { HttpError } from '../../../../core/http/HttpError.js';
import { PuterServer } from '../../../../server.js';
import { setupTestServer } from '../../../../testUtil.js';
import { withTestActor } from '../../shared/integrationTestUtil.js';
import { ChatCompletionDriver } from './ChatCompletionDriver.js';
import { AzureChatProvider } from './providers/azure/AzureChatProvider.js';
import { FakeChatProvider } from './providers/FakeChatProvider.js';
import { InfronProvider } from './providers/infron/InfronProvider.js';
import { NeuralwattProvider } from './providers/neuralwatt/NeuralwattProvider.js';
import { OpenAiChatProvider } from './providers/openai/OpenAiChatCompletionsProvider.js';
import { OpenRouterProvider } from './providers/openrouter/OpenRouterProvider.js';
import { TogetherAIProvider } from './providers/together/TogetherAIProvider.js';
import { XAI_CHAT_MODELS } from './providers/xai/catalog.js';
import { XAIProvider } from './providers/xai/XAIProvider.js';
import { handle_completion_output } from './utils/OpenAIUtil.js';
import { AzureChatProvider } from '../../providers/azure/chat/AzureChatProvider.js';
import { FakeChatProvider } from '../../providers/fake/chat/FakeChatProvider.js';
import { InfronProvider } from '../../providers/infron/chat/InfronProvider.js';
import { NeuralwattProvider } from '../../providers/neuralwatt/chat/NeuralwattProvider.js';
import { OpenAiChatProvider } from '../../providers/openai/chat/OpenAiChatCompletionsProvider.js';
import { OpenRouterProvider } from '../../providers/openrouter/chat/OpenRouterProvider.js';
import { TogetherAIProvider } from '../../providers/together/chat/TogetherAIProvider.js';
import { XAI_CHAT_MODELS } from '../../providers/xai/chat/catalog.js';
import { XAIProvider } from '../../providers/xai/chat/XAIProvider.js';
import { handle_completion_output } from '../../protocols/openaiCompatible/OpenAIUtil.js';
import { clearUnhealthyRoutes, isRouteUnhealthy } from './utils/providerHealth.js';
let server: PuterServer;
@@ -0,0 +1,715 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* Puter is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with Puter. If not, see <https://www.gnu.org/licenses/>.
*/
/**
* Seeded fuzzing of `ChatCompletionDriver.complete` through every registered
* provider. The vendors are played at the HTTP boundary, so the real SDKs and
* the providers' own request and response code run.
*/
import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest';
import { isHttpError } from '../../../../core/http/HttpError.js';
import type { PuterServer } from '../../../../server.js';
import { setupTestServer } from '../../../../testUtil.js';
import { kv } from '../../../../util/kvSingleton.js';
import { isDriverStreamResult } from '../../../meta.js';
import {
corruptReply,
Findings,
fuzzCases,
hostileValue,
mutate,
pick,
type Random,
seededRandom,
unclassifiedFailure,
upstream,
type UpstreamReply,
type UpstreamRequest,
} from '../../shared/fuzzTestUtil.js';
import {
readStreamResult,
withTestActor,
} from '../../shared/integrationTestUtil.js';
import { ChatCompletionDriver } from './ChatCompletionDriver.js';
import type { ICompleteArguments } from './types.js';
import { clearUnhealthyRoutes } from './utils/providerHealth.js';
vi.mock('undici', async (importOriginal) => ({
...(await importOriginal<typeof import('undici')>()),
fetch: (input: unknown, init?: RequestInit) => upstream.fetch(input, init),
}));
// groq-sdk brings its own HTTP client. Groq's API is OpenAI-compatible, so the
// OpenAI SDK stands in for it and the requests reach the stub like the rest.
vi.mock('groq-sdk', async () => ({ default: (await import('openai')).OpenAI }));
vi.mock('axios', () => {
const request = (config: Parameters<typeof upstream.axios>[0]) =>
upstream.axios(config);
const get = (url: string, config?: object) =>
upstream.axios({ ...config, url, method: 'GET' });
return { default: { request, get }, request, get };
});
// -- The vendors, played at the HTTP boundary --
const TEXT = 'pong';
const chatUsage = { prompt_tokens: 3, completion_tokens: 5, total_tokens: 8 };
const chatChunk = (delta: object, finish: string | null) => ({
id: 'chatcmpl-1',
object: 'chat.completion.chunk',
created: 1,
model: 'm',
choices: [{ index: 0, delta, finish_reason: finish }],
});
const responsesMessage = {
type: 'message',
id: 'msg_1',
role: 'assistant',
status: 'completed',
content: [{ type: 'output_text', text: TEXT, annotations: [] }],
};
const responsesBody = {
id: 'resp_1',
object: 'response',
status: 'completed',
model: 'm',
output: [responsesMessage],
usage: {
input_tokens: 3,
output_tokens: 5,
total_tokens: 8,
input_tokens_details: { cached_tokens: 0 },
output_tokens_details: { reasoning_tokens: 0 },
},
};
const anthropicUsage = { input_tokens: 3, output_tokens: 5 };
const geminiBody = {
candidates: [
{
index: 0,
content: { role: 'model', parts: [{ text: TEXT }] },
finishReason: 'STOP',
},
],
usageMetadata: {
promptTokenCount: 3,
candidatesTokenCount: 5,
totalTokenCount: 8,
},
};
const anthropicEvents = [
{
type: 'message_start',
message: {
id: 'msg_1',
type: 'message',
role: 'assistant',
model: 'm',
content: [],
stop_reason: null,
stop_sequence: null,
usage: { input_tokens: 3, output_tokens: 1 },
},
},
{
type: 'content_block_start',
index: 0,
content_block: { type: 'text', text: '' },
},
{
type: 'content_block_delta',
index: 0,
delta: { type: 'text_delta', text: TEXT },
},
{ type: 'content_block_stop', index: 0 },
{
type: 'message_delta',
delta: { stop_reason: 'end_turn', stop_sequence: null },
usage: { output_tokens: 5 },
},
{ type: 'message_stop' },
];
const responsesEvents = [
{
type: 'response.created',
response: { ...responsesBody, status: 'in_progress', output: [] },
},
{
type: 'response.output_item.added',
output_index: 0,
item: { ...responsesMessage, status: 'in_progress', content: [] },
},
{
type: 'response.output_text.delta',
item_id: 'msg_1',
output_index: 0,
content_index: 0,
delta: TEXT,
},
{
type: 'response.output_item.done',
output_index: 0,
item: responsesMessage,
},
{ type: 'response.completed', response: responsesBody },
];
/** A well-formed answer from whichever vendor API the request went to. */
const healthyUpstream = ({
url,
json,
}: UpstreamRequest): UpstreamReply | undefined => {
const path = url.pathname;
// A caller-supplied media link that leads nowhere.
if (url.hostname === 'upstream.test') return { status: 404, body: '' };
const streaming = (json as { stream?: unknown } | undefined)?.stream;
if (path.endsWith('/chat/completions')) {
if (!streaming) {
return {
body: {
id: 'chatcmpl-1',
object: 'chat.completion',
created: 1,
model: 'm',
choices: [
{
index: 0,
message: { role: 'assistant', content: TEXT },
finish_reason: 'stop',
},
],
usage: chatUsage,
},
};
}
return {
sse: [
{ data: chatChunk({ role: 'assistant', content: TEXT }, null) },
{ data: chatChunk({}, 'stop') },
{
data: {
...chatChunk({}, null),
choices: [],
usage: chatUsage,
},
},
{ data: '[DONE]' },
],
};
}
if (path.endsWith('/responses')) {
return streaming
? {
sse: responsesEvents.map((data) => ({
event: data.type,
data,
})),
}
: { body: responsesBody };
}
if (path.endsWith('/messages/count_tokens')) {
return { body: { input_tokens: 3 } };
}
if (path.endsWith('/messages')) {
return streaming
? {
sse: anthropicEvents.map((data) => ({
event: data.type,
data,
})),
}
: {
body: {
id: 'msg_1',
type: 'message',
role: 'assistant',
model: 'm',
content: [{ type: 'text', text: TEXT }],
stop_reason: 'end_turn',
stop_sequence: null,
usage: anthropicUsage,
},
};
}
if (path.endsWith(':generateContent')) return { body: geminiBody };
if (path.endsWith(':streamGenerateContent')) {
return { sse: [{ data: geminiBody }] };
}
if (path.endsWith(':countTokens')) return { body: { totalTokens: 3 } };
if (path.endsWith('/moderations')) {
return {
body: {
id: 'modr_1',
model: 'm',
results: [
{ flagged: false, categories: {}, category_scores: {} },
],
},
};
}
return undefined;
};
// -- Requests to mutate --
const PNG =
'data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8z8BQDwAEhQGAhKmMIQAAAABJRU5ErkJggg==';
const weatherTool = {
type: 'function',
function: {
name: 'get_weather',
description: 'Get the weather for a city',
parameters: {
type: 'object',
properties: { city: { type: 'string' } },
required: ['city'],
},
},
};
const CORPUS: Record<string, unknown>[] = [
{ messages: [{ role: 'user', content: 'hi' }] },
{
messages: [
{ role: 'system', content: 'Be brief.' },
{
role: 'user',
content: [
{ type: 'text', text: 'What is this?' },
{ type: 'image_url', image_url: { url: PNG } },
],
},
],
max_tokens: 64,
temperature: 0.5,
top_p: 0.9,
stream: true,
},
{
messages: [
{ role: 'user', content: 'Weather in Paris?' },
{
role: 'assistant',
content: null,
tool_calls: [
{
id: 'call_1',
type: 'function',
function: {
name: 'get_weather',
arguments: '{"city":"Paris"}',
},
},
],
},
{ role: 'tool', tool_call_id: 'call_1', content: 'Sunny' },
],
tools: [weatherTool],
tool_choice: 'auto',
parallel_tool_calls: false,
},
{
messages: [
{
role: 'user',
content: [
{
type: 'text',
text: 'Weather in Paris?',
cache_control: { type: 'ephemeral' },
},
],
},
{
role: 'assistant',
content: [
{ type: 'text', text: 'Checking.' },
{
type: 'tool_use',
id: 'toolu_1',
name: 'get_weather',
input: { city: 'Paris' },
},
],
},
{
role: 'user',
content: [
{
type: 'tool_result',
tool_use_id: 'toolu_1',
content: 'Sunny',
},
],
},
],
tools: [weatherTool, { type: 'web_search' }],
tool_choice: { type: 'function', function: { name: 'get_weather' } },
stream: true,
streamToolInput: true,
},
{
messages: [{ role: 'user', content: 'Reply in JSON.' }],
outputFormat: {
type: 'json_schema',
schema: { type: 'object', properties: { a: { type: 'string' } } },
},
reasoning: { effort: 'low' },
reasoning_effort: 'low',
thinking: { type: 'enabled', budget_tokens: 1024 },
text: { verbosity: 'low' },
verbosity: 'low',
stopSequences: ['END'],
topK: 5,
normalize: true,
},
{
messages: [{ role: 'user', content: 'hi' }],
instructions: 'Be brief.',
compaction: { trigger_tokens: 150000 },
context_management: { edits: [{ type: 'clear_tool_uses_20250919' }] },
cacheControl: { type: 'ephemeral', ttl: '1h' },
prompt_cache_key: 'k',
prompt_cache_retention: '24h',
serviceSpeed: 'fast',
service_tier: 'flex',
speed: 'fast',
metadata: { trace: 't' },
include: ['reasoning.encrypted_content'],
store: false,
truncation: 'auto',
taskBudget: { total: 1000, remaining: 500 },
safeguards: [{ type: 'prompt_injection' }],
anthropicBetas: ['context-management-2025-06-27'],
moderation: true,
response: { normalize: true },
fallback: false,
},
];
CORPUS.push(
{
messages: [
{
role: 'user',
content: [
{ type: 'text', text: 'Describe these.' },
{ type: 'image_url', image_url: PNG },
{
type: 'image',
source: {
type: 'base64',
media_type: 'image/png',
data: PNG.split(',')[1],
},
},
{
type: 'video_url',
video_url: { url: 'data:video/mp4;base64,AAAA' },
},
{
type: 'file',
file: {
filename: 'a.pdf',
file_data: 'data:application/pdf;base64,JVBERi0=',
},
},
{
type: 'document',
source: {
type: 'base64',
media_type: 'application/pdf',
data: 'JVBERi0=',
},
},
],
},
],
},
{
messages: [
{
role: 'user',
content: [
{ type: 'text', text: 'Read it.' },
{ puter_path: '~/Documents/a.png' },
],
},
],
},
// Model output sent back, in each vendor's shape.
{
messages: [
{ role: 'user', content: 'Think, then search.' },
{
role: 'assistant',
content: [
{ type: 'thinking', thinking: 'Hm.', signature: 'sig' },
{ type: 'redacted_thinking', data: 'x' },
{
type: 'reasoning',
id: 'rs_1',
summary: [{ type: 'summary_text', text: 'Hm.' }],
encrypted_content: 'enc',
},
{ type: 'text', text: 'Searching.' },
{
type: 'server_tool_use',
id: 'srvtoolu_1',
name: 'web_search',
input: { query: 'x' },
},
{
type: 'web_search_tool_result',
tool_use_id: 'srvtoolu_1',
content: [],
},
{
type: 'compaction',
id: 'cmp_1',
encrypted_content: 'enc',
},
],
reasoning: 'Hm.',
refusal: null,
},
{ type: 'compaction', id: 'cmp_2', encrypted_content: 'enc' },
{ role: 'user', content: 'Go on.' },
],
tools: [
{ type: 'web_search_20250305', name: 'web_search', max_uses: 3 },
{
type: 'function',
function: {
name: 'strict_tool',
strict: true,
parameters: {
type: 'object',
properties: {},
additionalProperties: false,
},
},
},
{
name: 'anthropic_style',
description: 'x',
input_schema: { type: 'object', properties: {} },
},
],
tool_choice: 'required',
thinking: { type: 'adaptive' },
prompt_cache_options: { ttl: '30m' },
instructions: [{ role: 'system', content: 'Be brief.' }],
},
);
const OPTION_NAMES = [
...new Set(CORPUS.flatMap((request) => Object.keys(request))),
'previous_response_id',
'conversation',
'prompt',
'prompt_cache_options',
'background',
'custom',
'customLimitMessage',
'test_mode',
];
// -- Harness --
const key = { apiKey: 'fuzz-key' };
const providers = {
claude: key,
'azure-openai': { ...key, apiURL: 'https://azure.upstream.test/openai' },
'openai-completion': key,
gemini: key,
meta: key,
groq: key,
deepseek: key,
mistral: key,
xai: key,
moonshot: key,
minimax: key,
zai: key,
alibaba: key,
byteplus: key,
hoonify: key,
ollama: { enabled: false },
};
let server: PuterServer;
let driver: ChatCompletionDriver;
// Every provider gets the same share of cases, whatever its catalog size.
let routes: Map<string, { provider: string; id: string }[]>;
beforeAll(async () => {
upstream.install();
server = await setupTestServer();
driver = new ChatCompletionDriver(
{ providers } as never,
server.clients,
server.stores,
server.services,
);
await driver.onServerStart();
routes = new Map();
for (const model of await driver.models()) {
if (!model.provider || model.provider === 'fake-chat') continue;
const route = { provider: model.provider, id: model.id };
routes.set(route.provider, [
...(routes.get(route.provider) ?? []),
route,
]);
}
}, 120_000);
afterAll(async () => {
await server?.shutdown();
upstream.uninstall();
});
const pickRoute = (random: Random) =>
pick(random, routes.get(pick(random, [...routes.keys()]))!);
/** Runs one completion to its end, stream included, and returns its failure. */
const complete = async (args: Record<string, unknown>): Promise<unknown> => {
try {
const result = await withTestActor(() =>
driver.complete(
structuredClone(args) as unknown as ICompleteArguments,
),
);
if (isDriverStreamResult(result)) await readStreamResult(result);
return undefined;
} catch (error) {
return error;
}
};
describe('ChatCompletionDriver seeded fuzzing', () => {
it('covers every configured provider', () => {
expect([...routes.keys()].sort()).toEqual([
'alibaba',
'azure-openai',
'azure-openai-responses',
'byteplus',
'claude',
'deepseek',
'gemini',
'groq',
'hoonify',
'meta',
'minimax',
'mistral',
'moonshotai',
'openai-completion',
'openai-responses',
'xai',
'zai',
]);
});
it('answers or rejects a malformed request without failing a healthy route', async () => {
upstream.respond = healthyUpstream;
upstream.unanswered.clear();
clearUnhealthyRoutes();
const random = seededRandom(0xc4a7);
const findings = new Findings();
for (let index = 0; index < fuzzCases(3000); index++) {
const route = pickRoute(random);
const args: Record<string, unknown> = {
...mutate(pick(random, CORPUS), random, OPTION_NAMES),
provider: route.provider,
model: route.id,
};
// Now and then the route itself is the malformed part.
if (random() < 0.05) {
args[pick(random, ['provider', 'model'])] =
hostileValue(random);
}
const label = `${route.provider}:${route.id}`;
const error = await complete(args);
const unclassified = error && unclassifiedFailure(error);
if (unclassified) findings.add(unclassified, index, label, args);
else if (isHttpError(error) && error.statusCode >= 500) {
const attempts = (error.fields?.attempts ?? []) as {
status?: number;
error: string;
}[];
findings.add(
`${error.statusCode} ${error.legacyCode}: ${error.message} [${attempts.map((a) => `${a.status ?? 'no status'} ${a.error}`).join(' | ')}]`,
index,
label,
args,
);
}
// The vendors answered every request, so no route failed.
for (const marked of kv.keys('aiChat:unhealthyRoute:*')) {
findings.add(`marked unhealthy ${marked}`, index, label, args);
}
clearUnhealthyRoutes();
}
for (const request of upstream.unanswered) {
findings.add(`no upstream fixture for ${request}`, -1, '', {});
}
findings.assertNone();
}, 600_000);
it('ends every request with an answer or a classified error when the upstream misbehaves', async () => {
const random = seededRandom(0x0b5e);
const findings = new Findings();
for (let index = 0; index < fuzzCases(2000); index++) {
const route = pickRoute(random);
const replies: unknown[] = [];
upstream.respond = (request) => {
const healthy = healthyUpstream(request);
const reply = healthy && corruptReply(healthy, random);
replies.push(reply);
return reply;
};
const error = await complete({
messages: [{ role: 'user', content: 'hi' }],
provider: route.provider,
model: route.id,
stream: random() < 0.5,
fallback: random() < 0.5,
});
const unclassified = error && unclassifiedFailure(error);
if (unclassified) {
findings.add(
unclassified,
index,
`${route.provider}:${route.id}`,
replies,
);
}
clearUnhealthyRoutes();
}
findings.assertNone();
}, 600_000);
});
@@ -38,11 +38,11 @@ import {
vi,
} from 'vitest';
import { PuterServer } from '../../server.js';
import { setupTestServer } from '../../testUtil.js';
import { withTestActor } from '../integrationTestUtil.js';
import { PuterServer } from '../../../../server.js';
import { setupTestServer } from '../../../../testUtil.js';
import { withTestActor } from '../../shared/integrationTestUtil.js';
import { ChatCompletionDriver } from './ChatCompletionDriver.js';
import { FakeChatProvider } from './providers/FakeChatProvider.js';
import { FakeChatProvider } from '../../providers/fake/chat/FakeChatProvider.js';
import type { IChatModel, ICompleteArguments } from './types.js';
// The puter_path resolver reads the user's filesystem; stub it and assert on
@@ -51,7 +51,7 @@ const { processPuterPathUploadsMock } = vi.hoisted(() => ({
processPuterPathUploadsMock: vi.fn(async (_messages: unknown) => {}),
}));
vi.mock('./providers/openai/fileUpload.js', () => ({
vi.mock('../../providers/openai/chat/fileUpload.js', () => ({
processPuterPathUploads: processPuterPathUploadsMock,
MAX_FILE_SIZE: 5 * 1_000_000,
}));
@@ -38,14 +38,14 @@ import {
vi,
} from 'vitest';
import { Context } from '../../core/context.js';
import type { UsageInput } from '../../services/metering/types.js';
import { PuterServer } from '../../server.js';
import { setupTestServer } from '../../testUtil.js';
import { withTestActor } from '../integrationTestUtil.js';
import { Context } from '../../../../core/context.js';
import type { UsageInput } from '../../../../services/metering/types.js';
import { PuterServer } from '../../../../server.js';
import { setupTestServer } from '../../../../testUtil.js';
import { withTestActor } from '../../shared/integrationTestUtil.js';
import { ChatCompletionDriver } from './ChatCompletionDriver.js';
import { clearUnhealthyRoutes, isRouteUnhealthy } from './utils/providerHealth.js';
import { FakeChatProvider } from './providers/FakeChatProvider.js';
import { FakeChatProvider } from '../../providers/fake/chat/FakeChatProvider.js';
import type { IChatCompleteResult } from './types.js';
import { SDK_TIMEOUT_MS } from './utils/sdkClient.js';
import type { AIChatStream } from './utils/Streaming.js';
@@ -44,21 +44,21 @@ import type { Request } from 'express';
import {
renderAnthropicError,
renderOpenAIError,
} from '../../controllers/puterai/wireErrors.js';
import { Context } from '../../core/context.js';
import { HttpError } from '../../core/http/HttpError.js';
import { PuterServer } from '../../server.js';
import { setupTestServer } from '../../testUtil.js';
import { withTestActor } from '../integrationTestUtil.js';
} from '../../../../controllers/puterai/wireErrors.js';
import { Context } from '../../../../core/context.js';
import { HttpError } from '../../../../core/http/HttpError.js';
import { PuterServer } from '../../../../server.js';
import { setupTestServer } from '../../../../testUtil.js';
import { withTestActor } from '../../shared/integrationTestUtil.js';
import { ChatCompletionDriver } from './ChatCompletionDriver.js';
import { OPEN_AI_MODELS } from './providers/openai/models.js';
import { OPEN_AI_MODELS } from '../../providers/openai/chat/models.js';
import {
clearUnhealthyRoutes,
isRouteUnhealthy,
markRouteUnhealthy,
} from './utils/providerHealth.js';
import { InfronProvider } from './providers/infron/InfronProvider.js';
import { UpstreamDeadlineError } from '../util/upstreamErrors.js';
import { InfronProvider } from '../../providers/infron/chat/InfronProvider.js';
import { UpstreamDeadlineError } from '../../../util/upstreamErrors.js';
// -- OpenAI SDK mock ------------------------------------------------
// Gemini reaches Google through `new openai.OpenAI()` (default export) and
@@ -44,17 +44,17 @@ import {
vi,
} from 'vitest';
import { Context } from '../../core/context.js';
import { HttpError } from '../../core/http/HttpError.js';
import { PuterServer } from '../../server.js';
import { setupTestServer } from '../../testUtil.js';
import { withTestActor } from '../integrationTestUtil.js';
import { Context } from '../../../../core/context.js';
import { HttpError } from '../../../../core/http/HttpError.js';
import { PuterServer } from '../../../../server.js';
import { setupTestServer } from '../../../../testUtil.js';
import { withTestActor } from '../../shared/integrationTestUtil.js';
import { COUNT_TOKENS, ChatCompletionDriver } from './ChatCompletionDriver.js';
import { ClaudeProvider } from './providers/claude/ClaudeProvider.js';
import { FakeChatProvider } from './providers/FakeChatProvider.js';
import { OllamaChatProvider } from './providers/ollama/OllamaProvider.js';
import { XAI_CHAT_MODELS } from './providers/xai/catalog.js';
import { XAIProvider } from './providers/xai/XAIProvider.js';
import { ClaudeProvider } from '../../providers/claude/chat/ClaudeProvider.js';
import { FakeChatProvider } from '../../providers/fake/chat/FakeChatProvider.js';
import { OllamaChatProvider } from '../../providers/ollama/chat/OllamaProvider.js';
import { XAI_CHAT_MODELS } from '../../providers/xai/chat/catalog.js';
import { XAIProvider } from '../../providers/xai/chat/XAIProvider.js';
import type {
IChatCompleteResult,
IChatProvider,
@@ -2396,3 +2396,97 @@ describe('ChatCompletionDriver service speed', () => {
},
);
});
describe('ChatCompletionDriver.complete option shapes', () => {
const hi = [{ role: 'user', content: 'hi' }];
it.each(['model', 'provider', 'service_tier'])(
'rejects a value that cannot be text as `%s`',
async (name) => {
const args = { model: 'fake', messages: hi, [name]: { toString: null } };
await expect(
withTestActor(() => driver.complete(args as never)),
).rejects.toMatchObject({
statusCode: 400,
legacyCode: 'bad_request',
message: `\`${name}\` is not a usable value`,
});
await expect(
withTestActor(() => driver[COUNT_TOKENS](args as never)),
).rejects.toMatchObject({ statusCode: 400 });
},
);
it('routes by the model when `provider` is an object, as it did before', async () => {
const completeSpy = vi.spyOn(FakeChatProvider.prototype, 'complete');
await withTestActor(() =>
driver.complete({
model: 'fake',
messages: hi,
provider: { order: ['Anthropic'] },
} as never),
);
expect(completeSpy).toHaveBeenCalledTimes(1);
});
it.each([false, 0, '', null])(
'treats `tools: %j` as no tools when counting tokens',
async (tools) => {
const countTokens = vi.fn(async (_args: object) => 1);
const fake = FakeChatProvider.prototype as {
countTokens?: typeof countTokens;
};
fake.countTokens = countTokens;
try {
await withTestActor(() =>
driver[COUNT_TOKENS]({
model: 'fake',
messages: hi,
tools,
} as never),
);
} finally {
delete fake.countTokens;
}
expect(countTokens.mock.calls[0]![0]).not.toHaveProperty('tools');
},
);
it.each([false, 0, '', null])(
'treats `tools: %j` as no tools',
async (tools) => {
const completeSpy = vi.spyOn(FakeChatProvider.prototype, 'complete');
await withTestActor(() =>
driver.complete({ model: 'fake', messages: hi, tools } as never),
);
expect(completeSpy.mock.calls[0]![0]).not.toHaveProperty('tools');
},
);
it('reports zero tokens in the cost event for a usage count that is not a number', async () => {
vi.spyOn(FakeChatProvider.prototype, 'models').mockResolvedValueOnce([
{
id: 'priced',
aliases: [],
costs_currency: 'usd-cents',
costs: { input_tokens: 1000, output_tokens: 2000 },
max_tokens: 8192,
},
]);
const d = await makeDriver();
const costEvents = captureEvent('ai.prompt.cost-calculated');
vi.spyOn(FakeChatProvider.prototype, 'complete').mockResolvedValueOnce({
message: { role: 'assistant', content: [{ type: 'text', text: 'ok' }] },
usage: { input_tokens: 10, output_tokens: { toString: null } },
finish_reason: 'stop',
} as never);
await withTestActor(() => d.complete({ model: 'priced', messages: hi }));
expect(costEvents[0]![1]).toMatchObject({
input_tokens: 10,
output_tokens: 0,
total_ucents: 10 * 1000,
});
});
});
@@ -19,56 +19,56 @@
import crypto from 'node:crypto';
import { PassThrough } from 'node:stream';
import { EventMap } from '../../clients/event/types.js';
import type { Actor } from '../../core/actor.js';
import { Context } from '../../core/context.js';
import { HttpError, isHttpError } from '../../core/http/HttpError.js';
import { insufficientCreditsError } from '../../services/metering/enforcement.js';
import { isFreeSubscription } from '../../services/metering/consts.js';
import type { MeteringService } from '../../services/metering/MeteringService.js';
import type { CreditHold } from '../../services/metering/types.js';
import { NO_CREDIT_HOLD } from '../../services/metering/types.js';
import type { DriverStreamResult } from '../meta.js';
import { PuterDriver } from '../types.js';
import { EventMap } from '../../../../clients/event/types.js';
import type { Actor } from '../../../../core/actor.js';
import { Context } from '../../../../core/context.js';
import { HttpError, isHttpError } from '../../../../core/http/HttpError.js';
import { insufficientCreditsError } from '../../../../services/metering/enforcement.js';
import { isFreeSubscription } from '../../../../services/metering/consts.js';
import type { MeteringService } from '../../../../services/metering/MeteringService.js';
import type { CreditHold } from '../../../../services/metering/types.js';
import { NO_CREDIT_HOLD } from '../../../../services/metering/types.js';
import type { DriverStreamResult } from '../../../meta.js';
import { PuterDriver } from '../../../types.js';
import {
type AiMeteringService,
factoredTotal,
requestAiCosts,
withAiCostFactor,
} from '../util/aiCostFactor.js';
import { AI_CONCURRENT, AI_RATE_LIMIT } from '../util/aiLimits.js';
} from '../../../../util/aiCostFactor.js';
import { AI_CONCURRENT, AI_RATE_LIMIT } from '../../shared/aiLimits.js';
import {
isCreditExhaustion as isUpstreamCreditExhaustion,
isUpstreamConnectionError,
isUpstreamTimeoutError,
sanitizeUpstreamMessage,
UpstreamDeadlineError,
} from '../util/upstreamErrors.js';
import { AlibabaProvider } from './providers/alibaba/AlibabaProvider.js';
import { AzureChatProvider } from './providers/azure/AzureChatProvider.js';
import { AzureResponsesProvider } from './providers/azure/AzureResponsesProvider.js';
import { BytePlusProvider } from './providers/byteplus/BytePlusProvider.js';
import { ClaudeProvider } from './providers/claude/ClaudeProvider.js';
import { DeepSeekProvider } from './providers/deepseek/DeepSeekProvider.js';
import { FakeChatProvider } from './providers/FakeChatProvider.js';
import { GeminiChatProvider } from './providers/gemini/GeminiChatProvider.js';
import { GroqAIProvider } from './providers/groq/GroqAIProvider.js';
import { HoonifyProvider } from './providers/hoonify/HoonifyProvider.js';
import { InfronProvider } from './providers/infron/InfronProvider.js';
import { MetaProvider } from './providers/meta/MetaProvider.js';
import { MiniMaxProvider } from './providers/minimax/MiniMaxProvider.js';
import { MistralAIProvider } from './providers/mistral/MistralAiProvider.js';
import { MoonshotProvider } from './providers/moonshot/MoonshotProvider.js';
import { NeuralwattProvider } from './providers/neuralwatt/NeuralwattProvider.js';
import { OllamaChatProvider } from './providers/ollama/OllamaProvider.js';
import type { ChatProviderConfig } from './providers/OpenAICompatProvider.js';
import { processPuterPathUploads } from './providers/openai/fileUpload.js';
import { OpenAiChatProvider } from './providers/openai/OpenAiChatCompletionsProvider.js';
import { OpenAiResponsesChatProvider } from './providers/openai/OpenAiChatResponsesProvider.js';
import { OpenRouterProvider } from './providers/openrouter/OpenRouterProvider.js';
import { TogetherAIProvider } from './providers/together/TogetherAIProvider.js';
import { XAIProvider } from './providers/xai/XAIProvider.js';
import { ZAIProvider } from './providers/zai/ZAIProvider.js';
} from '../../../util/upstreamErrors.js';
import { AlibabaProvider } from '../../providers/alibaba/chat/AlibabaProvider.js';
import { AzureChatProvider } from '../../providers/azure/chat/AzureChatProvider.js';
import { AzureResponsesProvider } from '../../providers/azure/chat/AzureResponsesProvider.js';
import { BytePlusProvider } from '../../providers/byteplus/chat/BytePlusProvider.js';
import { ClaudeProvider } from '../../providers/claude/chat/ClaudeProvider.js';
import { DeepSeekProvider } from '../../providers/deepseek/chat/DeepSeekProvider.js';
import { FakeChatProvider } from '../../providers/fake/chat/FakeChatProvider.js';
import { GeminiChatProvider } from '../../providers/gemini/chat/GeminiChatProvider.js';
import { GroqAIProvider } from '../../providers/groq/chat/GroqAIProvider.js';
import { HoonifyProvider } from '../../providers/hoonify/chat/HoonifyProvider.js';
import { InfronProvider } from '../../providers/infron/chat/InfronProvider.js';
import { MetaProvider } from '../../providers/meta/chat/MetaProvider.js';
import { MiniMaxProvider } from '../../providers/minimax/chat/MiniMaxProvider.js';
import { MistralAIProvider } from '../../providers/mistral/chat/MistralAiProvider.js';
import { MoonshotProvider } from '../../providers/moonshot/chat/MoonshotProvider.js';
import { NeuralwattProvider } from '../../providers/neuralwatt/chat/NeuralwattProvider.js';
import { OllamaChatProvider } from '../../providers/ollama/chat/OllamaProvider.js';
import type { ChatProviderConfig } from '../../protocols/openaiCompatible/OpenAICompatProvider.js';
import { processPuterPathUploads } from '../../providers/openai/chat/fileUpload.js';
import { OpenAiChatProvider } from '../../providers/openai/chat/OpenAiChatCompletionsProvider.js';
import { OpenAiResponsesChatProvider } from '../../providers/openai/chat/OpenAiChatResponsesProvider.js';
import { OpenRouterProvider } from '../../providers/openrouter/chat/OpenRouterProvider.js';
import { TogetherAIProvider } from '../../providers/together/chat/TogetherAIProvider.js';
import { XAIProvider } from '../../providers/xai/chat/XAIProvider.js';
import { ZAIProvider } from '../../providers/zai/chat/ZAIProvider.js';
import type {
IChatCompleteResult,
IChatMessageResult,
@@ -103,9 +103,13 @@ import {
normalizeResultToOpenAI,
shouldPresentAsOpenAI,
} from './utils/normalizeToOpenAI.js';
import { isToolChoice, toolChoiceFromWire } from './utils/openaiParams.js';
import {
catalogPrice,
isToolChoice,
toolChoiceFromWire,
} from '../../protocols/openaiCompatible/openaiParams.js';
import { assertScalarOptions } from '../../shared/scalarOptions.js';
import { catalogPrice } from '../../shared/catalogPrice.js';
import {
costKeys,
costsForRequest,
isFreeModel,
@@ -121,7 +125,7 @@ import {
TRANSIENT_UNHEALTHY_TTL_SEC,
UNHEALTHY_TTL_SEC,
} from './utils/providerHealth.js';
import { CATALOG_TTL_MS } from './utils/cachedRemoteCatalog.js';
import { CATALOG_TTL_MS } from '../../shared/cachedRemoteCatalog.js';
import { SDK_TIMEOUT_MS } from './utils/sdkClient.js';
import { fromFinishReason } from './utils/stopReason.js';
import { AIChatStream } from './utils/Streaming.js';
@@ -178,6 +182,9 @@ export const COUNT_TOKENS = Symbol('aiChat.countTokens');
*/
const HOLD_RENEW_INTERVAL_MS = 5 * 60 * 1000;
// Options that reach error messages and request headers as text.
const SCALAR_OPTIONS = ['model', 'provider', 'service_tier'];
/**
* A moderation refusal is a completion that was produced and charged, then
* withheld — not a route failure. Retrying it on a fallback provider would bill
@@ -585,7 +592,7 @@ export class ChatCompletionDriver extends PuterDriver {
readonly driverName = 'ai-chat';
readonly isDefault = true;
// Shared AI policy — see `drivers/util/aiLimits.ts` for the tier table.
// Shared AI policy — see `drivers/ai/shared/aiLimits.ts` for the tier table.
readonly rateLimit = AI_RATE_LIMIT;
readonly concurrent = AI_CONCURRENT;
@@ -690,6 +697,7 @@ export class ChatCompletionDriver extends PuterDriver {
{ legacyCode: 'invalid_service_speed' },
);
}
assertScalarOptions(args, SCALAR_OPTIONS);
const abortSignal = Context.get('abortSignal');
abortSignal?.throwIfAborted();
const actor = Context.get('actor');
@@ -744,6 +752,9 @@ export class ChatCompletionDriver extends PuterDriver {
args.messages = normalizeMediaParts(normalize_messages(args.messages));
if (args.tools) {
normalize_tools_object(args.tools);
} else {
// `tools: enabled && list` leaves `false` here; it means no tools.
delete args.tools;
}
// A raw `/drivers/call` caller still sends the OpenAI wire form
// ('auto'/'none'/'required', {type:'function', function:{name}}) —
@@ -1490,6 +1501,7 @@ export class ChatCompletionDriver extends PuterDriver {
async [COUNT_TOKENS](
args: ICompleteArguments,
): Promise<{ input_tokens: number }> {
assertScalarOptions(args, SCALAR_OPTIONS);
let intendedProvider = args.provider || '';
if (!args.model && !intendedProvider) {
intendedProvider = 'azure-openai';
@@ -1515,6 +1527,9 @@ export class ChatCompletionDriver extends PuterDriver {
args.messages = normalizeMediaParts(normalize_messages(args.messages));
if (args.tools) {
normalize_tools_object(args.tools);
} else {
// `tools: enabled && list` leaves `false` here; it means no tools.
delete args.tools;
}
const provider = this.#providers[model.provider!];
@@ -1848,20 +1863,20 @@ export class ChatCompletionDriver extends PuterDriver {
const cost = usageCost(usage, model, usageCosts, pricingTime);
const { inputKey, outputKey } = costKeys(model);
// The vendor reports usage, and pricing skips a count that is not a
// number; so does this.
const count = (value: unknown) =>
typeof value === 'number' && Number.isFinite(value) ? value : 0;
const inputTokens = cost
? Number(
usage[inputKey] ??
usage.prompt_tokens ??
usage.input_tokens ??
0,
? count(
usage[inputKey] ?? usage.prompt_tokens ?? usage.input_tokens,
)
: 0;
const outputTokens = cost
? Number(
? count(
usage[outputKey] ??
usage.completion_tokens ??
usage.output_tokens ??
0,
usage.output_tokens,
)
: 0;
const inputMicroCents = cost?.inputMicroCents ?? 0;
@@ -47,29 +47,29 @@
* on that test.)
*/
import { readdirSync } from 'node:fs';
import { existsSync, readdirSync } from 'node:fs';
import { dirname, join } from 'node:path';
import { fileURLToPath, pathToFileURL } from 'node:url';
import { describe, expect, it } from 'vitest';
import type { IChatModel } from '../types.js';
import { ALIBABA_MODELS } from './alibaba/models.js';
import { AZURE_MODELS } from './azure/models.js';
import { BYTEPLUS_MODELS } from './byteplus/models.js';
import { CLAUDE_MODELS } from './claude/models.js';
import { DEEPSEEK_MODELS } from './deepseek/models.js';
import { GEMINI_MODELS } from './gemini/models.js';
import { GROQ_MODELS } from './groq/models.js';
import { HOONIFY_MODELS } from './hoonify/models.js';
import { META_MODELS } from './meta/models.js';
import { MINIMAX_MODELS } from './minimax/models.js';
import { MISTRAL_MODELS } from './mistral/models.js';
import { MOONSHOT_MODELS } from './moonshot/models.js';
import { OPEN_AI_MODELS } from './openai/models.js';
import { OPEN_ROUTER_MODEL_OVERRIDES } from './openrouter/modelOverrides.js';
import { XAI_MODELS } from './xai/models.js';
import { ZAI_MODELS } from './zai/models.js';
import type { IChatModel } from './types.js';
import { ALIBABA_MODELS } from '../../providers/alibaba/chat/models.js';
import { AZURE_MODELS } from '../../providers/azure/chat/models.js';
import { BYTEPLUS_MODELS } from '../../providers/byteplus/chat/models.js';
import { CLAUDE_MODELS } from '../../providers/claude/chat/models.js';
import { DEEPSEEK_MODELS } from '../../providers/deepseek/chat/models.js';
import { GEMINI_MODELS } from '../../providers/gemini/chat/models.js';
import { GROQ_MODELS } from '../../providers/groq/chat/models.js';
import { HOONIFY_MODELS } from '../../providers/hoonify/chat/models.js';
import { META_MODELS } from '../../providers/meta/chat/models.js';
import { MINIMAX_MODELS } from '../../providers/minimax/chat/models.js';
import { MISTRAL_MODELS } from '../../providers/mistral/chat/models.js';
import { MOONSHOT_MODELS } from '../../providers/moonshot/chat/models.js';
import { OPEN_AI_MODELS } from '../../providers/openai/chat/models.js';
import { OPEN_ROUTER_MODEL_OVERRIDES } from '../../providers/openrouter/chat/modelOverrides.js';
import { XAI_MODELS } from '../../providers/xai/chat/models.js';
import { ZAI_MODELS } from '../../providers/zai/chat/models.js';
// Providers whose catalog is fetched at runtime (OpenRouter, Ollama, Together,
// Infron, Neuralwatt) have nothing static to check and are absent by design;
@@ -201,28 +201,34 @@ describe('CATALOGS', () => {
// silent gap these tests are about, so the list is checked against what
// is actually on disk.
//
// The scan covers every non-test source file under providers/*/ with
// "model" in its name — models.ts, but also siblings like openrouter's
// modelOverrides.ts. Importing every provider file regardless of name
// would drag in SDK modules for a filename sweep, so that naming
// convention is the one assumption left unenforced here: a catalog in a
// file named without "model" would escape this net.
it('lists every static catalog under providers/', async () => {
const here = dirname(fileURLToPath(import.meta.url));
// The scan covers every non-test source file under providers/*/chat/
// with "model" in its name — models.ts, but also siblings like
// openrouter's modelOverrides.ts. Importing every provider file
// regardless of name would drag in SDK modules for a filename sweep, so
// that naming convention is the one assumption left unenforced here: a
// catalog in a file named without "model" would escape this net.
it('lists every static catalog under providers/*/chat/', async () => {
const providersDir = join(
dirname(fileURLToPath(import.meta.url)),
'../../providers',
);
const registered = new Map(CATALOGS);
const found = new Set<string>();
const problems: string[] = [];
for (const dir of readdirSync(here, { withFileTypes: true })) {
for (const dir of readdirSync(providersDir, { withFileTypes: true })) {
if (!dir.isDirectory()) continue;
const chatDir = join(providersDir, dir.name, 'chat');
if (!existsSync(chatDir)) continue;
for (const file of readdirSync(join(here, dir.name))) {
for (const file of readdirSync(chatDir)) {
if (!/model/i.test(file)) continue;
if (!file.endsWith('.ts') || file.endsWith('.test.ts')) {
continue;
}
const module = await import(
pathToFileURL(join(here, dir.name, file)).href
pathToFileURL(join(chatDir, file)).href
);
for (const [name, value] of Object.entries(module)) {
// A catalog is a non-empty array of entries carrying an
@@ -233,16 +239,19 @@ describe('CATALOGS', () => {
value.length > 0 &&
typeof value[0]?.id === 'string';
if (!isCatalog) continue;
found.add(name);
if (!registered.has(name)) {
problems.push(`${dir.name}/${file} exports ${name}`);
problems.push(
`${dir.name}/chat/${file} exports ${name}`,
);
} else if (registered.get(name) !== value) {
// The row's label names this export but its value is
// a different array — the wrong catalog would be the
// one getting checked.
problems.push(
`the CATALOGS row named ${name} does not hold ` +
`the ${name} that ${dir.name}/${file} exports`,
`the ${name} that ${dir.name}/chat/${file} exports`,
);
}
}
@@ -250,5 +259,7 @@ describe('CATALOGS', () => {
}
expect(problems, problems.join('\n')).toEqual([]);
// A scan pointed at the wrong directory finds nothing and passes.
expect([...found].sort()).toEqual([...registered.keys()].sort());
});
});
@@ -39,41 +39,41 @@
import { describe, expect, it, vi } from 'vitest';
import type { MeteringService } from '../../../services/metering/MeteringService.js';
import { withTestActor } from '../../integrationTestUtil.js';
import type { MeteringService } from '../../../../services/metering/MeteringService.js';
import { withTestActor } from '../../shared/integrationTestUtil.js';
import type {
IChatMessageResult,
IChatModel,
IChatProvider,
} from '../types.js';
} from './types.js';
import {
needsOpenAICoercion,
normalizeResultToOpenAI,
} from '../utils/normalizeToOpenAI.js';
} from './utils/normalizeToOpenAI.js';
import { AlibabaProvider } from './alibaba/AlibabaProvider.js';
import { AzureChatProvider } from './azure/AzureChatProvider.js';
import { AzureResponsesProvider } from './azure/AzureResponsesProvider.js';
import { BytePlusProvider } from './byteplus/BytePlusProvider.js';
import { ClaudeProvider } from './claude/ClaudeProvider.js';
import { DeepSeekProvider } from './deepseek/DeepSeekProvider.js';
import { FakeChatProvider } from './FakeChatProvider.js';
import { GeminiChatProvider } from './gemini/GeminiChatProvider.js';
import { GroqAIProvider } from './groq/GroqAIProvider.js';
import { HoonifyProvider } from './hoonify/HoonifyProvider.js';
import { InfronProvider } from './infron/InfronProvider.js';
import { MetaProvider } from './meta/MetaProvider.js';
import { MiniMaxProvider } from './minimax/MiniMaxProvider.js';
import { MistralAIProvider } from './mistral/MistralAiProvider.js';
import { MoonshotProvider } from './moonshot/MoonshotProvider.js';
import { NeuralwattProvider } from './neuralwatt/NeuralwattProvider.js';
import { OllamaChatProvider } from './ollama/OllamaProvider.js';
import { OpenAiChatProvider } from './openai/OpenAiChatCompletionsProvider.js';
import { OpenAiResponsesChatProvider } from './openai/OpenAiChatResponsesProvider.js';
import { OpenRouterProvider } from './openrouter/OpenRouterProvider.js';
import { TogetherAIProvider } from './together/TogetherAIProvider.js';
import { XAIProvider } from './xai/XAIProvider.js';
import { ZAIProvider } from './zai/ZAIProvider.js';
import { AlibabaProvider } from '../../providers/alibaba/chat/AlibabaProvider.js';
import { AzureChatProvider } from '../../providers/azure/chat/AzureChatProvider.js';
import { AzureResponsesProvider } from '../../providers/azure/chat/AzureResponsesProvider.js';
import { BytePlusProvider } from '../../providers/byteplus/chat/BytePlusProvider.js';
import { ClaudeProvider } from '../../providers/claude/chat/ClaudeProvider.js';
import { DeepSeekProvider } from '../../providers/deepseek/chat/DeepSeekProvider.js';
import { FakeChatProvider } from '../../providers/fake/chat/FakeChatProvider.js';
import { GeminiChatProvider } from '../../providers/gemini/chat/GeminiChatProvider.js';
import { GroqAIProvider } from '../../providers/groq/chat/GroqAIProvider.js';
import { HoonifyProvider } from '../../providers/hoonify/chat/HoonifyProvider.js';
import { InfronProvider } from '../../providers/infron/chat/InfronProvider.js';
import { MetaProvider } from '../../providers/meta/chat/MetaProvider.js';
import { MiniMaxProvider } from '../../providers/minimax/chat/MiniMaxProvider.js';
import { MistralAIProvider } from '../../providers/mistral/chat/MistralAiProvider.js';
import { MoonshotProvider } from '../../providers/moonshot/chat/MoonshotProvider.js';
import { NeuralwattProvider } from '../../providers/neuralwatt/chat/NeuralwattProvider.js';
import { OllamaChatProvider } from '../../providers/ollama/chat/OllamaProvider.js';
import { OpenAiChatProvider } from '../../providers/openai/chat/OpenAiChatCompletionsProvider.js';
import { OpenAiResponsesChatProvider } from '../../providers/openai/chat/OpenAiChatResponsesProvider.js';
import { OpenRouterProvider } from '../../providers/openrouter/chat/OpenRouterProvider.js';
import { TogetherAIProvider } from '../../providers/together/chat/TogetherAIProvider.js';
import { XAIProvider } from '../../providers/xai/chat/XAIProvider.js';
import { ZAIProvider } from '../../providers/zai/chat/ZAIProvider.js';
// ── Upstream SDK mocks ──────────────────────────────────────────────
// One create-mock per wire dialect; every provider speaking that dialect
@@ -18,7 +18,6 @@
*/
import { HttpError } from '@heyputer/backend/src/core/http';
import { claudeToolPolicy } from '../providers/claude/anthropicPolicy.js';
// Anthropic server/typed tool ids (`web_search_20250305`,
// `advisor_20260301`, …) and the OpenAI Responses `web_search` tool pass
@@ -124,7 +123,7 @@ export const normalize_tools_object = (tools) => {
};
// Claude-only extras, carried on the `{type:'function', ...}`
// wrapper so `make_claude_tools` can forward them; a top-level
// wrapper so `claudeToolPolicy` can forward them; a top-level
// `strict` (Claude's own convention) moves onto `function.strict`.
const extras = {};
if (tool.cache_control !== undefined)
@@ -169,76 +168,3 @@ export const normalize_tools_object = (tools) => {
}
return tools;
};
// OpenAI's own tool conventions pass through untouched.
const OPENAI_NATIVE_WEB_SEARCH = new Set(['web_search', 'web_search_preview']);
const ANTHROPIC_WEB_SEARCH_TYPE = (type) =>
type === 'web_search_20250305' || /^web_search_2026/.test(type ?? '');
/**
* Converts a normalized tools object to the format expected by an OpenAI-family
* dialect: function tools drop their Anthropic-only extras (`cache_control`,
* `defer_loading`, `eager_input_streaming`, `input_examples`); every other
* Anthropic typed/server tool (`web_search_20250305`, `advisor_20260301`,
* `bash_*`, …) is dropped, since none of it is a wire shape OpenAI recognizes.
* The one exception: an Anthropic web-search tool is translated to OpenAI's own
* `web_search` tool for the Responses dialect, which supports it natively.
*
* @param {any} tools
* @param {{ dialect?: 'chat' | 'responses' }} [opts]
* @returns {any[] | undefined}
*/
export const make_openai_tools = (tools, opts = {}) => {
if (!tools) return tools;
const dialect = opts.dialect ?? 'chat';
const out = [];
for (const tool of tools) {
if (!tool || typeof tool !== 'object') continue;
if (tool.type === 'function') {
const stripped = { ...tool };
delete stripped.cache_control;
delete stripped.defer_loading;
delete stripped.eager_input_streaming;
delete stripped.input_examples;
out.push(stripped);
continue;
}
if (OPENAI_NATIVE_WEB_SEARCH.has(tool.type)) {
out.push(tool);
continue;
}
if (dialect === 'responses' && ANTHROPIC_WEB_SEARCH_TYPE(tool.type)) {
out.push({
type: 'web_search',
...(tool.allowed_domains
? { filters: { allowed_domains: tool.allowed_domains } }
: {}),
...(tool.user_location
? { user_location: tool.user_location }
: {}),
});
continue;
}
// Any other Anthropic typed/server tool (web_search_* on the chat
// dialect, advisor, bash, memory, web_fetch, tool_search, …) has no
// OpenAI equivalent — drop it rather than forward an unrecognized type.
}
return out;
};
/**
* This function will convert a normalized tools object to the format expected
* by Claude: function tools become `{name, description, input_schema, ...}`,
* and typed/server tools (`web_search`, `web_search_20250305`,
* `advisor_20260301`, …) go through the allowlist/clamp/reject policy in
* `claudeToolPolicy` instead of being treated as malformed function
* definitions.
*
* @param {any} tools
* @param {{ models?: import('../types.js').IChatModel[] }} [opts]
* @returns {any[] | undefined}
*/
export const make_claude_tools = (tools, opts = {}) => {
if (!tools) return undefined;
return claudeToolPolicy(tools, opts.models ?? []).tools;
};
@@ -20,13 +20,11 @@
import { describe, expect, it } from 'vitest';
// @ts-expect-error — sibling JS module without an adjacent .d.ts
import {
make_claude_tools,
make_openai_tools,
normalize_json_schema,
normalize_tools_object,
} from './FunctionCalling.js';
// All four exports are pure data transforms — these tests just feed
// Both exports are pure data transforms — these tests just feed
// inputs and check the normalized output, no service mocks involved.
// ── normalize_json_schema ───────────────────────────────────────────
@@ -242,159 +240,3 @@ describe('normalize_tools_object', () => {
expect(out[0].function.parameters.properties).toBe('x');
});
});
// ── make_openai_tools ───────────────────────────────────────────────
describe('make_openai_tools', () => {
it('keeps a function tool, dropping its Anthropic-only extras', () => {
const tools = [
{
type: 'function',
function: { name: 'lookup', parameters: { type: 'object' } },
cache_control: { type: 'ephemeral' },
defer_loading: true,
eager_input_streaming: true,
input_examples: [{}],
},
];
expect(make_openai_tools(tools)).toEqual([
{
type: 'function',
function: { name: 'lookup', parameters: { type: 'object' } },
},
]);
});
it('drops an Anthropic web_search tool on the chat dialect', () => {
const tools = [{ type: 'web_search_20250305', name: 'web_search' }];
expect(make_openai_tools(tools, { dialect: 'chat' })).toEqual([]);
});
it('converts an Anthropic web_search tool to OpenAI web_search on the responses dialect', () => {
const tools = [
{
type: 'web_search_20250305',
name: 'web_search',
max_uses: 5,
allowed_domains: ['example.com'],
user_location: { type: 'approximate' },
},
];
expect(
make_openai_tools(tools, { dialect: 'responses' }),
).toEqual([
{
type: 'web_search',
filters: { allowed_domains: ['example.com'] },
user_location: { type: 'approximate' },
},
]);
});
it('keeps OpenAI web_search/web_search_preview tools verbatim', () => {
const tools = [{ type: 'web_search' }, { type: 'web_search_preview' }];
expect(make_openai_tools(tools, { dialect: 'responses' })).toEqual(
tools,
);
});
it('drops other Anthropic typed/server tools (advisor, bash, …)', () => {
const tools = [
{ type: 'advisor_20260301', model: 'claude-haiku-4-5' },
{ type: 'bash_20250124', name: 'bash' },
];
expect(make_openai_tools(tools, { dialect: 'responses' })).toEqual([]);
});
it('passes a falsy tools value through unchanged', () => {
expect(make_openai_tools(undefined)).toBeUndefined();
expect(make_openai_tools(null)).toBeNull();
});
});
describe('make_claude_tools with a malformed tool', () => {
it.each([
['type:function with no function property', { type: 'function' }],
['a null entry', null],
['a bare string entry', 'lookup'],
])('answers 400 for a tool with %s', (_label, tool) => {
// A TypeError here has no status, so the retry loop reads it as a
// provider failure and marks the route unhealthy for every caller.
expect(() => make_claude_tools([tool] as never)).toThrowError(
expect.objectContaining({
statusCode: 400,
legacyCode: 'bad_request',
}),
);
});
it('answers 400 for an unresolvable advisor model', () => {
expect(() =>
make_claude_tools([
{ type: 'advisor_20260301', model: 'not-a-real-model' },
] as never),
).toThrowError(
expect.objectContaining({ statusCode: 400, legacyCode: 'bad_request' }),
);
});
it('answers 400 for an mcp_toolset tool', () => {
expect(() =>
make_claude_tools([{ type: 'mcp_toolset' }] as never),
).toThrowError(
expect.objectContaining({ statusCode: 400, legacyCode: 'bad_request' }),
);
});
});
// ── make_claude_tools ───────────────────────────────────────────────
describe('make_claude_tools', () => {
it('returns undefined when tools is undefined', () => {
expect(make_claude_tools(undefined)).toBeUndefined();
});
it('returns [] when tools is an empty array', () => {
expect(make_claude_tools([])).toEqual([]);
});
it('flattens the OpenAI {function:{...}} wrapper into Claude shape', () => {
const out = make_claude_tools([
{
type: 'function',
function: {
name: 'lookup',
description: 'Search the docs',
parameters: {
type: 'object',
properties: { q: { type: 'string' } },
},
},
},
]);
expect(out).toEqual([
{
name: 'lookup',
description: 'Search the docs',
input_schema: {
type: 'object',
properties: { q: { type: 'string' } },
},
},
]);
});
it('maps the OpenAI {type:"web_search"} form to web_search_20250305', () => {
const out = make_claude_tools([{ type: 'web_search' }] as never);
expect(out).toEqual([
{ type: 'web_search_20250305', name: 'web_search', max_uses: 10 },
]);
});
it('passes a typed server tool through by its own regexp-matched type', () => {
const out = make_claude_tools([
{ type: 'bash_20250124', name: 'bash' },
] as never);
expect(out).toEqual([{ type: 'bash_20250124', name: 'bash' }]);
});
});
@@ -18,6 +18,7 @@
*/
import { HttpError } from '@heyputer/backend/src/core/http';
import { convertsToText } from '../../../shared/scalarOptions.js';
/**
* Normalizes a single message into a standardized format with role and content
@@ -154,6 +155,39 @@ export const normalize_single_message = (message, params = {}) => {
}
}
// Providers write a block's type and a tool call's id into text, which
// throws on a value that can't be converted.
for (const block of message.content) {
if (!convertsToText(block.type)) {
throw new HttpError(
400,
'each message content item type must be a string',
{ legacyCode: 'bad_request' },
);
}
const callId =
block.type === 'tool_use'
? block.id
: block.type === 'tool_result'
? block.tool_use_id
: undefined;
if (!convertsToText(callId)) {
throw new HttpError(400, 'a tool call id must be a string', {
legacyCode: 'bad_request',
});
}
// A tool's result is text or content blocks; any other object is sent
// as its JSON, like the content of a `tool` message above.
if (
block.type === 'tool_result' &&
block.content &&
typeof block.content === 'object' &&
!Array.isArray(block.content)
) {
block.content = JSON.stringify(block.content);
}
}
// Remove "text" properties from content blocks with type=tool_use
for (let i = 0; i < message.content.length; i++) {
if (message.content[i].type !== 'tool_use') {
@@ -449,3 +449,106 @@ describe('extract_text', () => {
).toThrow(expect.objectContaining({ statusCode: 400 }));
});
});
describe('normalize_single_message block validation', () => {
it.each([
[
'an assistant tool call',
{
role: 'assistant',
tool_calls: [
{ id: { toString: null }, function: { name: 'f', arguments: '{}' } },
],
},
],
[
'a tool_use block',
{
role: 'assistant',
content: [
{
type: 'tool_use',
id: { toString: null },
name: 'f',
input: {},
},
],
},
],
[
'a tool message',
{ role: 'tool', tool_call_id: [{ toString: null }], content: 'ok' },
],
[
'a tool_result block',
{
role: 'user',
content: [
{
type: 'tool_result',
tool_use_id: { toString: null },
content: 'ok',
},
],
},
],
])('rejects a tool call id that cannot be text on %s', (_label, message) => {
expect(() => normalize_single_message(message)).toThrow(
expect.objectContaining({
statusCode: 400,
message: 'a tool call id must be a string',
}),
);
});
it('keeps a numeric tool call id as the caller sent it', () => {
const message = normalize_single_message({
role: 'assistant',
content: [{ type: 'tool_use', id: 7, name: 'f', input: {} }],
});
expect(message.content[0].id).toBe(7);
});
it('keeps a tool call that has no id', () => {
const message = normalize_single_message({
role: 'assistant',
tool_calls: [{ function: { name: 'f', arguments: '{}' } }],
});
expect(message.content).toEqual([
{ type: 'tool_use', id: undefined, name: 'f', input: '{}' },
]);
});
it('rejects a content item whose type cannot be text', () => {
expect(() =>
normalize_single_message({
role: 'user',
content: [{ type: { toString: null }, text: 'hi' }],
}),
).toThrow(
expect.objectContaining({
statusCode: 400,
message: 'each message content item type must be a string',
}),
);
});
it('sends an object tool result as its JSON', () => {
const message = normalize_single_message({
role: 'user',
content: [
{ type: 'tool_result', tool_use_id: 't1', content: { temp: 21 } },
],
});
expect(message.content[0].content).toBe('{"temp":21}');
});
it('keeps a tool result made of content blocks', () => {
const blocks = [{ type: 'text', text: 'Sunny' }];
const message = normalize_single_message({
role: 'user',
content: [{ type: 'tool_result', tool_use_id: 't1', content: blocks }],
});
expect(message.content[0].content).toEqual(blocks);
});
});
@@ -18,20 +18,21 @@
*/
/**
* Inline-compaction translation helpers.
* Inline-compaction opt-in helpers.
*
* The driver-facing surface exposes a single provider-neutral opt-in
* (`compaction: boolean | { trigger_tokens }` on `ICompleteArguments`), plus a
* raw `context_management` escape hatch for callers hitting `/responses` with
* the OpenAI-native shape. These helpers map that neutral opt-in to each
* provider's SDK shape so the providers stay free of opt-in-parsing logic.
* the OpenAI-native shape. This module reads the opt-in; each dialect maps it
* to its own SDK shape (`toOpenAiContextManagement` in the OpenAI protocol,
* `toAnthropicContextManagement` in the Claude provider).
*/
/**
* @param {boolean | { trigger_tokens?: number } | undefined} compaction
* @returns {{ enabled: boolean; trigger_tokens?: number }}
*/
const readCompaction = (compaction) => {
export const readCompaction = (compaction) => {
if (compaction === true) return { enabled: true };
if (compaction && typeof compaction === 'object') {
return {
@@ -44,65 +45,6 @@ const readCompaction = (compaction) => {
return { enabled: false };
};
/**
* Build OpenAI Responses `context_management` from the neutral opt-in. A raw
* `context_management` passthrough (already in OpenAI shape) wins.
*
* @param {{
* compaction?: boolean | { trigger_tokens?: number };
* context_management?: unknown;
* }} args
* @returns {{ type: 'compaction'; compact_threshold?: number }[] | undefined}
*/
export const toOpenAiContextManagement = (args) => {
if (args.context_management !== undefined) {
return /** @type {any} */ (args.context_management);
}
const { enabled, trigger_tokens } = readCompaction(args.compaction);
if (!enabled) return undefined;
return [
{
type: 'compaction',
...(trigger_tokens !== undefined
? { compact_threshold: trigger_tokens }
: {}),
},
];
};
/**
* Build Anthropic `context_management` (beta `compact-2026-01-12`) from the
* neutral opt-in. A raw `context_management` passthrough wins.
*
* @param {{
* compaction?: boolean | { trigger_tokens?: number };
* context_management?: unknown;
* }} args
* @returns {{ edits: Record<string, unknown>[] } | undefined}
*/
export const toAnthropicContextManagement = (args) => {
if (args.context_management !== undefined) {
return /** @type {any} */ (args.context_management);
}
const { enabled, trigger_tokens } = readCompaction(args.compaction);
if (!enabled) return undefined;
return {
edits: [
{
type: 'compact_20260112',
...(trigger_tokens !== undefined
? {
trigger: {
type: 'input_tokens',
value: trigger_tokens,
},
}
: {}),
},
],
};
};
/**
* Whether the request opted into inline compaction by any route.
*
@@ -0,0 +1,62 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { describe, expect, it } from 'vitest';
import { messagesHaveCompaction, wantsCompaction } from './compaction.js';
describe('wantsCompaction', () => {
it('is false without opt-in', () => {
expect(wantsCompaction({})).toBe(false);
expect(wantsCompaction({ compaction: false })).toBe(false);
});
it('is true for the neutral opt-in or a raw passthrough', () => {
expect(wantsCompaction({ compaction: true })).toBe(true);
expect(wantsCompaction({ compaction: { trigger_tokens: 1 } })).toBe(
true,
);
expect(wantsCompaction({ context_management: [] })).toBe(true);
});
});
describe('messagesHaveCompaction', () => {
it('detects a round-tripped compaction content block', () => {
expect(
messagesHaveCompaction([
{ role: 'user', content: [{ type: 'text', text: 'hi' }] },
{
role: 'assistant',
content: [
{ type: 'compaction', encrypted_content: 'ENC' },
],
},
]),
).toBe(true);
});
it('is false for ordinary messages or non-arrays', () => {
expect(
messagesHaveCompaction([
{ role: 'user', content: [{ type: 'text', text: 'hi' }] },
]),
).toBe(false);
expect(messagesHaveCompaction(undefined)).toBe(false);
expect(messagesHaveCompaction('nope')).toBe(false);
});
});
@@ -40,7 +40,7 @@ type Part = Record<string, unknown>;
const isObject = (value: unknown): value is Part =>
!!value && typeof value === 'object' && !Array.isArray(value);
export { parseDataUri } from '../../util/dataUri.js';
export { parseDataUri } from '../../../../util/dataUri.js';
/** The URL out of a bare string or an OpenAI-style `{ url }` object. */
export const mediaUrlOf = (value: unknown): string | undefined => {
@@ -18,19 +18,19 @@
*/
import { describe, expect, it } from 'vitest';
import type { Actor } from '../../../core/actor.js';
import { ALIBABA_MODELS } from '../providers/alibaba/models.js';
import { BYTEPLUS_MODELS } from '../providers/byteplus/models.js';
import { DEEPSEEK_MODELS } from '../providers/deepseek/models.js';
import { GROQ_MODELS } from '../providers/groq/models.js';
import { HOONIFY_MODELS } from '../providers/hoonify/models.js';
import { MINIMAX_MODELS } from '../providers/minimax/models.js';
import { MISTRAL_MODELS } from '../providers/mistral/models.js';
import { MOONSHOT_MODELS } from '../providers/moonshot/models.js';
import { ZAI_MODELS } from '../providers/zai/models.js';
import type { Actor } from '../../../../../core/actor.js';
import { ALIBABA_MODELS } from '../../../providers/alibaba/chat/models.js';
import { BYTEPLUS_MODELS } from '../../../providers/byteplus/chat/models.js';
import { DEEPSEEK_MODELS } from '../../../providers/deepseek/chat/models.js';
import { GROQ_MODELS } from '../../../providers/groq/chat/models.js';
import { HOONIFY_MODELS } from '../../../providers/hoonify/chat/models.js';
import { MINIMAX_MODELS } from '../../../providers/minimax/chat/models.js';
import { MISTRAL_MODELS } from '../../../providers/mistral/chat/models.js';
import { MOONSHOT_MODELS } from '../../../providers/moonshot/chat/models.js';
import { ZAI_MODELS } from '../../../providers/zai/chat/models.js';
import type { IChatModel } from '../types.js';
import { meterChatUsage } from './meterChatUsage.js';
import { splitCachedPrompt } from './OpenAIUtil.js';
import { splitCachedPrompt } from '../../../protocols/openaiCompatible/OpenAIUtil.js';
import { usageCost } from './pricing.js';
type Recorded = [Record<string, number>, string, Record<string, number>];
@@ -17,8 +17,8 @@
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import type { Actor } from '../../../core/actor.js';
import type { MeteringService } from '../../../services/metering/MeteringService.js';
import type { Actor } from '../../../../../core/actor.js';
import type { MeteringService } from '../../../../../services/metering/MeteringService.js';
import type { IChatModel, ServiceSpeed } from '../types.js';
import { serviceSpeedBillingModel } from './serviceSpeed.js';
import { buildCostsOverride } from './pricing.js';
@@ -19,7 +19,7 @@
*/
import { describe, expect, it } from 'vitest';
import { GEMINI_MODELS } from '../providers/gemini/models.js';
import { GEMINI_MODELS } from '../../../providers/gemini/chat/models.js';
import type { IChatModel } from '../types.js';
import {
compareModelPreference,
@@ -21,7 +21,6 @@ import { describe, expect, it } from 'vitest';
import type { IChatModel } from '../types.js';
import {
buildCostsOverride,
catalogPrice,
costsForRequest,
isFreeModel,
isOutputCostKey,
@@ -480,12 +479,6 @@ it('preserves free classification for legacy hyphenated token keys', () => {
expect(isFreeModel(model({ 'input-tokens': 0, 'output-tokens': 0 }))).toBe(true);
});
it('leaves overflowing catalog unit conversions unknown', () => {
expect(catalogPrice(1e308, 100)).toBeUndefined();
expect(catalogPrice(0, 100)).toBe(0);
expect(catalogPrice('0.0000000014', 100_000_000)).toBeCloseTo(0.14);
});
describe('costsForRequest prompt-size overrides', () => {
const model = (minPromptTokensInclusive?: boolean): IChatModel => ({
id: 'tiered',
@@ -23,29 +23,7 @@ import type {
ModelPricingOverride,
UsageDetails,
} from '../types.js';
/**
* Absent, negative and placeholder prices remain unpriced, including empty
* strings.
*/
export const catalogPrice = (
value: unknown,
multiplier = 1,
): number | undefined => {
if (typeof value !== 'number' && typeof value !== 'string')
return undefined;
if (
typeof value === 'string' &&
!/^(?:\d+(?:\.\d*)?|\.\d+)(?:e[+-]?\d+)?$/i.test(value.trim())
)
return undefined;
const price = Number(value);
if (!Number.isFinite(price) || price < 0) return undefined;
const converted = price * multiplier;
return Number.isFinite(converted) && !(price > 0 && converted === 0)
? converted
: undefined;
};
import { catalogPrice } from '../../../shared/catalogPrice.js';
const CENTS_PER_USD = 100;
const MTOK = 1_000_000;
@@ -360,7 +338,8 @@ export const buildCostsOverride = (
* What a usage object costs in µ¢, split into its input and output sides.
* `recordedCosts` are the per-key costs a provider already metered with and win
* over the model's rates, so the figure matches the ledger. Null when nothing
* in the usage could be priced. `now` is the instant the request was priced at.
* in the usage could be priced. `now` is the instant the request was priced
* at.
*/
export const usageCost = (
usage: Record<string, unknown>,
@@ -19,7 +19,7 @@
*/
import { afterEach, describe, expect, it } from 'vitest';
import { kv } from '../../../util/kvSingleton.js';
import { kv } from '../../../../../util/kvSingleton.js';
import {
clearUnhealthyRoutes,
isRouteUnhealthy,
@@ -30,7 +30,7 @@
* blip doesn't hand the model's traffic to a reseller for long.
*/
import { kv } from '../../../util/kvSingleton.js';
import { kv } from '../../../../../util/kvSingleton.js';
/**
* How long a route stays marked after a failure that won't heal on its own,
@@ -19,21 +19,21 @@
import { describe, expect, it } from 'vitest';
import type { IChatModel, ServiceSpeed } from '../types.js';
import { ALIBABA_MODELS } from '../providers/alibaba/models.js';
import { AZURE_MODELS } from '../providers/azure/models.js';
import { BYTEPLUS_MODELS } from '../providers/byteplus/models.js';
import { CLAUDE_MODELS } from '../providers/claude/models.js';
import { DEEPSEEK_MODELS } from '../providers/deepseek/models.js';
import { GEMINI_MODELS } from '../providers/gemini/models.js';
import { GROQ_MODELS } from '../providers/groq/models.js';
import { HOONIFY_MODELS } from '../providers/hoonify/models.js';
import { META_MODELS } from '../providers/meta/models.js';
import { MINIMAX_MODELS } from '../providers/minimax/models.js';
import { MISTRAL_MODELS } from '../providers/mistral/models.js';
import { MOONSHOT_MODELS } from '../providers/moonshot/models.js';
import { OPEN_AI_MODELS } from '../providers/openai/models.js';
import { XAI_MODELS } from '../providers/xai/models.js';
import { ZAI_MODELS } from '../providers/zai/models.js';
import { ALIBABA_MODELS } from '../../../providers/alibaba/chat/models.js';
import { AZURE_MODELS } from '../../../providers/azure/chat/models.js';
import { BYTEPLUS_MODELS } from '../../../providers/byteplus/chat/models.js';
import { CLAUDE_MODELS } from '../../../providers/claude/chat/models.js';
import { DEEPSEEK_MODELS } from '../../../providers/deepseek/chat/models.js';
import { GEMINI_MODELS } from '../../../providers/gemini/chat/models.js';
import { GROQ_MODELS } from '../../../providers/groq/chat/models.js';
import { HOONIFY_MODELS } from '../../../providers/hoonify/chat/models.js';
import { META_MODELS } from '../../../providers/meta/chat/models.js';
import { MINIMAX_MODELS } from '../../../providers/minimax/chat/models.js';
import { MISTRAL_MODELS } from '../../../providers/mistral/chat/models.js';
import { MOONSHOT_MODELS } from '../../../providers/moonshot/chat/models.js';
import { OPEN_AI_MODELS } from '../../../providers/openai/chat/models.js';
import { XAI_MODELS } from '../../../providers/xai/chat/models.js';
import { ZAI_MODELS } from '../../../providers/zai/chat/models.js';
import { shouldPresentAsOpenAI } from './normalizeToOpenAI.js';
import {
isServiceSpeedUnavailable,
@@ -20,19 +20,19 @@
import { describe, expect, it } from 'vitest';
import {
makeMeteringStub,
optionalEnv,
withTestActor,
} from '../../integrationTestUtil.js';
} from '../../../shared/integrationTestUtil.js';
import { optionalEnv } from '../../../../../testEnv.js';
import type {
IChatMessageResult,
IChatProvider,
ICompleteArguments,
} from '../types.js';
import { OpenAiResponsesChatProvider } from '../providers/openai/OpenAiChatResponsesProvider.js';
import { GeminiChatProvider } from '../providers/gemini/GeminiChatProvider.js';
import { MiniMaxProvider } from '../providers/minimax/MiniMaxProvider.js';
import { MoonshotProvider } from '../providers/moonshot/MoonshotProvider.js';
import { MistralAIProvider } from '../providers/mistral/MistralAiProvider.js';
import { OpenAiResponsesChatProvider } from '../../../providers/openai/chat/OpenAiChatResponsesProvider.js';
import { GeminiChatProvider } from '../../../providers/gemini/chat/GeminiChatProvider.js';
import { MiniMaxProvider } from '../../../providers/minimax/chat/MiniMaxProvider.js';
import { MoonshotProvider } from '../../../providers/moonshot/chat/MoonshotProvider.js';
import { MistralAIProvider } from '../../../providers/mistral/chat/MistralAiProvider.js';
import { selectServiceSpeed } from './serviceSpeed.js';
type Case = {
@@ -23,7 +23,7 @@ import { AIChatStream } from './Streaming.js';
import {
handle_completion_output,
handle_completion_output_responses_api,
} from './OpenAIUtil.js';
} from '../../../protocols/openaiCompatible/OpenAIUtil.js';
import type { IChatModel, ServiceSpeed } from '../types.js';
import {
requestedServiceSpeed,
@@ -33,10 +33,10 @@ import {
isServiceSpeedUnavailable,
} from './serviceSpeed.js';
import { meterChatUsage } from './meterChatUsage.js';
import { OPEN_AI_MODELS } from '../providers/openai/models.js';
import { XAI_MODELS } from '../providers/xai/models.js';
import { GEMINI_MODELS } from '../providers/gemini/models.js';
import { CLAUDE_MODELS } from '../providers/claude/models.js';
import { OPEN_AI_MODELS } from '../../../providers/openai/chat/models.js';
import { XAI_MODELS } from '../../../providers/xai/chat/models.js';
import { GEMINI_MODELS } from '../../../providers/gemini/chat/models.js';
import { CLAUDE_MODELS } from '../../../providers/claude/chat/models.js';
const model = (provider: string, id: string): IChatModel => ({
provider,
@@ -216,6 +216,24 @@ describe('estimatePromptTokens', () => {
});
});
describe('estimatePromptTokens compaction blocks', () => {
const withCompaction = (encrypted_content: unknown) => [
{ role: 'assistant', content: [{ type: 'compaction', encrypted_content }] },
];
it('counts the encrypted content as text', () => {
expect(estimatePromptTokens(withCompaction('a'.repeat(400)))).toBe(
estimateTextTokens('a'.repeat(400)),
);
});
it('counts content that is not a string by its JSON', () => {
expect(
estimatePromptTokens(withCompaction({ toString: null, blob: 'abcdefgh' })),
).toBe(estimateTextTokens('{"toString":null,"blob":"abcdefgh"}'));
});
});
describe('estimateToolTokens', () => {
it('is zero for no tools', () => {
expect(estimateToolTokens(undefined)).toBe(0);
@@ -33,10 +33,10 @@ import {
createWordScanBudget,
estimateTextTokens,
type WordScanBudget,
} from '../../util/tokenEstimate.js';
} from '../../../shared/tokenEstimate.js';
// Re-exported so existing consumers keep one import site for all estimators.
export { estimateTextTokens } from '../../util/tokenEstimate.js';
export { estimateTextTokens } from '../../../shared/tokenEstimate.js';
/** Tokens per byte of an inline (base64) attachment; see `attachmentTokens`. */
const ATTACHMENT_BYTES_PER_TOKEN = 150;
@@ -154,7 +154,12 @@ const partTokens = (part: unknown, budget: WordScanBudget): number => {
);
}
if (p.type === 'compaction') {
return estimateTextTokens(String(p.encrypted_content ?? ''), budget);
return estimateTextTokens(
typeof p.encrypted_content === 'string'
? p.encrypted_content
: JSON.stringify(p.encrypted_content ?? ''),
budget,
);
}
return UNMEASURED_ATTACHMENT_TOKENS;
@@ -41,23 +41,22 @@ import {
} from 'vitest';
import { v4 as uuidv4 } from 'uuid';
import { runWithContext } from '../../core/context.js';
import { type Actor, SYSTEM_ACTOR } from '../../core/actor.js';
import { PuterServer } from '../../server.js';
import { setupTestServer } from '../../testUtil.js';
import { withAiCostFactorHook } from '../integrationTestUtil.js';
import { generateDefaultFsentries } from '../../util/userProvisioning.js';
import { CLOUDFLARE_IMAGE_GENERATION_MODELS } from './providers/cloudflare/models.js';
import cloudflareModelSearchFixture from './providers/cloudflare/fixtures/model-search.json';
import { GEMINI_IMAGE_GENERATION_MODELS } from './providers/gemini/models.js';
import { OPEN_AI_IMAGE_GENERATION_MODELS } from './providers/openai/models.js';
import { REPLICATE_IMAGE_GENERATION_MODELS } from './providers/replicate/models.js';
import { TOGETHER_IMAGE_GENERATION_MODELS } from './providers/together/models.js';
import { XAI_IMAGE_GENERATION_MODELS } from './providers/xai/models.js';
import xaiLiveCatalog from './providers/xai/fixtures/image-generation-models.json';
import { runWithContext } from '../../../../core/context.js';
import { type Actor, SYSTEM_ACTOR } from '../../../../core/actor.js';
import { PuterServer } from '../../../../server.js';
import { setupTestServer, withAiCostFactorHook } from '../../../../testUtil.js';
import { generateDefaultFsentries } from '../../../../util/userProvisioning.js';
import { CLOUDFLARE_IMAGE_GENERATION_MODELS } from '../../providers/cloudflare/image/models.js';
import cloudflareModelSearchFixture from '../../providers/cloudflare/image/fixtures/model-search.json';
import { GEMINI_IMAGE_GENERATION_MODELS } from '../../providers/gemini/image/models.js';
import { OPEN_AI_IMAGE_GENERATION_MODELS } from '../../providers/openai/image/models.js';
import { REPLICATE_IMAGE_GENERATION_MODELS } from '../../providers/replicate/image/models.js';
import { TOGETHER_IMAGE_GENERATION_MODELS } from '../../providers/together/image/models.js';
import { XAI_IMAGE_GENERATION_MODELS } from '../../providers/xai/image/models.js';
import xaiLiveCatalog from '../../providers/xai/image/fixtures/image-generation-models.json';
import { ImageGenerationDriver } from './ImageGenerationDriver.js';
import { CATALOG_TTL_MS } from '../ai-chat/utils/cachedRemoteCatalog.js';
import { ReplicateImageGenerationProvider } from './providers/replicate/ReplicateImageGenerationProvider.js';
import { CATALOG_TTL_MS } from '../../shared/cachedRemoteCatalog.js';
import { ReplicateImageGenerationProvider } from '../../providers/replicate/image/ReplicateImageGenerationProvider.js';
// ── SDK mocks ──────────────────────────────────────────────────────
//
@@ -141,8 +140,8 @@ vi.mock('replicate', () => {
const { secureFetchMock } = vi.hoisted(() => ({ secureFetchMock: vi.fn() }));
vi.mock('../../util/secureHttp.js', async (importOriginal) => ({
...(await importOriginal<typeof import('../../util/secureHttp.js')>()),
vi.mock('../../../../util/secureHttp.js', async (importOriginal) => ({
...(await importOriginal<typeof import('../../../../util/secureHttp.js')>()),
secureFetch: secureFetchMock,
}));
@@ -483,6 +482,20 @@ describe('ImageGenerationDriver.generate provider routing', () => {
},
);
it('rejects an input image type that is not a string before any provider runs', async () => {
await expect(withActor(() =>
driver.generate({
model: 'gpt-image-2',
prompt: 'make it blue',
input_image: 'data:image/png;base64,aGVsbG8=',
input_image_mime_type: 5,
} as never),
)).rejects.toMatchObject({
statusCode: 400,
message: 'input_image_mime_type must be a string',
});
});
it('rejects an excluded model regardless of caller casing', async () => {
await expect(withActor(() =>
driver.generate({
@@ -1273,7 +1286,7 @@ describe('ImageGenerationDriver.generate audit log', () => {
// ── puter_output_path ─────────────────────────────────────────────
describe('ImageGenerationDriver.generate puter_output_path', () => {
const TEST_ACTOR: import('../../core/actor.js').Actor = {
const TEST_ACTOR: import('../../../../core/actor.js').Actor = {
user: { uuid: 'a1b2c3d4-e5f6-4a7b-8c9d-0e1f2a3b4c5d', id: 42, username: 'testuser' },
};
@@ -1462,7 +1475,7 @@ describe('ImageGenerationDriver.generate puter_output_path', () => {
});
it('throws 400 when actor has no user ID but puter_output_path is set', async () => {
const noIdActor: import('../../core/actor.js').Actor = {
const noIdActor: import('../../../../core/actor.js').Actor = {
user: { uuid: 'f0e1d2c3-b4a5-4968-8777-0a1b2c3d4e5f', username: 'noone' },
};
await expect(
@@ -19,32 +19,32 @@
import { assertImagePrompt } from './imageValidation.js';
import crypto from 'node:crypto';
import { Context } from '../../core/context.js';
import { HttpError } from '../../core/http/HttpError.js';
import type { Actor } from '../../core/actor.js';
import { PuterDriver } from '../types.js';
import { Context } from '../../../../core/context.js';
import { HttpError } from '../../../../core/http/HttpError.js';
import type { Actor } from '../../../../core/actor.js';
import { PuterDriver } from '../../../types.js';
import {
type AiMeteringService,
withAiCostFactor,
} from '../util/aiCostFactor.js';
import { AI_CONCURRENT, AI_RATE_LIMIT } from '../util/aiLimits.js';
} from '../../../../util/aiCostFactor.js';
import { AI_CONCURRENT, AI_RATE_LIMIT } from '../../shared/aiLimits.js';
import {
resolveOutputPath,
saveGeneratedMediaToFS,
} from '../util/generatedMedia.js';
import { readPuterImageInputs } from '../util/mediaInput.js';
import { ModelCatalog } from '../util/modelCatalog.js';
import { CATALOG_TTL_MS } from '../ai-chat/utils/cachedRemoteCatalog.js';
import { readProviderKey } from '../util/providerRegistry.js';
import { BytePlusImageProvider } from './providers/byteplus/BytePlusImageProvider.js';
import { CloudflareImageProvider } from './providers/cloudflare/CloudflareImageProvider.js';
import { GeminiImageProvider } from './providers/gemini/GeminiImageProvider.js';
import { KlingImageProvider } from './providers/kling/KlingImageProvider.js';
import { OpenAiImageProvider } from './providers/openai/OpenAiImageProvider.js';
import { ReplicateImageGenerationProvider } from './providers/replicate/ReplicateImageGenerationProvider.js';
} from '../../shared/generatedMedia.js';
import { readPuterImageInputs } from '../../shared/mediaInput.js';
import { ModelCatalog } from '../../shared/modelCatalog.js';
import { CATALOG_TTL_MS } from '../../shared/cachedRemoteCatalog.js';
import { readProviderKey } from '../../../../util/aiProviderKey.js';
import { BytePlusImageProvider } from '../../providers/byteplus/image/BytePlusImageProvider.js';
import { CloudflareImageProvider } from '../../providers/cloudflare/image/CloudflareImageProvider.js';
import { GeminiImageProvider } from '../../providers/gemini/image/GeminiImageProvider.js';
import { KlingImageProvider } from '../../providers/kling/image/KlingImageProvider.js';
import { OpenAiImageProvider } from '../../providers/openai/image/OpenAiImageProvider.js';
import { ReplicateImageGenerationProvider } from '../../providers/replicate/image/ReplicateImageGenerationProvider.js';
import { resolveImageSize } from './imageDimensions.js';
import { TogetherImageProvider } from './providers/together/TogetherImageProvider.js';
import { XAIImageProvider } from './providers/xai/XAIImageProvider.js';
import { TogetherImageProvider } from '../../providers/together/image/TogetherImageProvider.js';
import { XAIImageProvider } from '../../providers/xai/image/XAIImageProvider.js';
import type { IGenerateParams, IImageModel, IImageProvider } from './types.js';
import { assertInputImagesShape } from './inputImage.js';
@@ -78,7 +78,7 @@ export class ImageGenerationDriver extends PuterDriver {
];
readonly isDefault = true;
// Shared AI policy — see `drivers/util/aiLimits.ts` for the tier table.
// Shared AI policy — see `drivers/ai/shared/aiLimits.ts` for the tier table.
readonly rateLimit = AI_RATE_LIMIT;
readonly concurrent = AI_CONCURRENT;
@@ -177,7 +177,11 @@ export class ImageGenerationDriver extends PuterDriver {
// settled here — once — before any provider runs.
assertImagePrompt(args.prompt);
assertInputImagesShape(args, 'image generation');
for (const option of ['quality', 'resolution'] as const) {
for (const option of [
'quality',
'resolution',
'input_image_mime_type',
] as const) {
if (args[option] != null && typeof args[option] !== 'string') {
throw new HttpError(400, `${option} must be a string`, {
legacyCode: 'bad_request',
@@ -17,7 +17,7 @@
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { HttpError } from '../../core/http/HttpError.js';
import { HttpError } from '../../../../core/http/HttpError.js';
import type {
IGenerateParams,
IImageModel,
@@ -17,7 +17,7 @@
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { SVG_SNIFF_WINDOW, sniffImageMime } from '../../util/appIcon.js';
import { SVG_SNIFF_WINDOW, sniffImageMime } from '../../../../util/appIcon.js';
// Base64 chars needed to recover SVG_SNIFF_WINDOW decoded bytes.
const SNIFF_BASE64_CHARS = Math.ceil((SVG_SNIFF_WINDOW * 4) / 3);
@@ -17,7 +17,7 @@
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { HttpError } from '../../core/http/HttpError.js';
import { HttpError } from '../../../../core/http/HttpError.js';
export function assertImagePrompt(prompt: unknown): asserts prompt is string {
if (typeof prompt !== 'string' || !prompt.trim()) {
@@ -37,7 +37,7 @@ import {
const { secureFetchMock } = vi.hoisted(() => ({ secureFetchMock: vi.fn() }));
vi.mock('../../util/secureHttp.js', () => ({ secureFetch: secureFetchMock }));
vi.mock('../../../../util/secureHttp.js', () => ({ secureFetch: secureFetchMock }));
const fetchResponse = (
body: Buffer,
@@ -21,12 +21,12 @@
* Shared helpers for `input_images` (image-to-image) handling across image
* providers. `input_images` is the canonical, cross-provider field; an entry
* may be a public URL, a data-URI, or raw base64. The string-level helpers live
* in `drivers/util/imageInput.ts` so video providers share them; this module
* re-exports them and adds the `IGenerateParams`-shaped validation.
* in `drivers/ai/shared/imageInput.ts` so video providers share them; this
* module re-exports them and adds the `IGenerateParams`-shaped validation.
*/
import { HttpError } from '../../core/http/HttpError.js';
import { assertInputImageString } from '../util/imageInput.js';
import { HttpError } from '../../../../core/http/HttpError.js';
import { assertInputImageString } from '../../shared/imageInput.js';
import type { IGenerateParams } from './types.js';
/** Most input images one request may carry. */
@@ -41,7 +41,7 @@ export {
parseDataUri,
toBase64DataUri,
toUrlOrDataUri,
} from '../util/imageInput.js';
} from '../../shared/imageInput.js';
/**
* Validate `input_image` / `input_images` once, where the driver call arrives.
@@ -0,0 +1,396 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* Puter is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with Puter. If not, see <https://www.gnu.org/licenses/>.
*/
/**
* Seeded fuzzing of the image, video, speech and OCR drivers through every
* provider they register. One suite covers the six so the server boots once.
* Every vendor refuses the request, so what runs is each driver's validation,
* each provider's request building, and the mapping of the refusal.
*/
import { PollyClient } from '@aws-sdk/client-polly';
import { TextractClient } from '@aws-sdk/client-textract';
import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest';
import type { Actor } from '../../../core/actor.js';
import type { PuterServer } from '../../../server.js';
import { setupTestServer } from '../../../testUtil.js';
import { isDriverStreamResult } from '../../meta.js';
import {
fuzzCalls,
fuzzCases,
hostileValue,
pick,
type Random,
upstream,
} from '../shared/fuzzTestUtil.js';
import {
readStreamResult,
withTestActor,
} from '../shared/integrationTestUtil.js';
vi.mock('undici', async (importOriginal) => ({
...(await importOriginal<typeof import('undici')>()),
fetch: (input: unknown, init?: RequestInit) => upstream.fetch(input, init),
}));
vi.mock('axios', () => {
const request = (config: Parameters<typeof upstream.axios>[0]) =>
upstream.axios(config);
const get = (url: string, config?: object) =>
upstream.axios({ ...config, url, method: 'GET' });
return { default: { request, get }, request, get };
});
const PNG =
'data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8z8BQDwAEhQGAhKmMIQAAAABJRU5ErkJggg==';
const AUDIO =
'data:audio/wav;base64,UklGRiQAAABXQVZFZm10IBAAAAABAAEAQB8AAIA+AAACABAAZGF0YQAAAAA=';
const PDF = 'data:application/pdf;base64,JVBERi0xLjQK';
type Route = { provider: string; model?: string };
interface Target {
name: string;
driver: string;
method: string;
/** The providers the suite must reach, so a lost registration fails it. */
providers: string[];
routes: (driver: Record<string, any>) => Promise<Route[]>;
corpus: Record<string, unknown>[];
}
const modelRoutes = async (driver: Record<string, any>): Promise<Route[]> =>
(await driver.models()).map((model: { provider: string; id: string }) => ({
provider: model.provider,
model: model.id,
}));
const providerRoutes = async (driver: Record<string, any>): Promise<Route[]> =>
(await driver.list()).map((provider: string) => ({ provider }));
const TARGETS: Target[] = [
{
name: 'image generation',
driver: 'aiImage',
method: 'generate',
providers: [
'byteplus-image-generation',
'cloudflare-image-generation',
'gemini-image-generation',
'kling-image-generation',
'openai-image-generation',
'replicate-image-generation',
'xai-image-generation',
],
routes: modelRoutes,
corpus: [
{ prompt: 'a red cube' },
{
prompt: 'a red cube',
ratio: { w: 1024, h: 768 },
quality: 'high',
seed: 7,
negative_prompt: 'blur',
steps: 4,
guidance: 2,
response_format: 'png',
output_megapixels: '1',
providerOptions: { go_fast: true },
},
{
prompt: 'make it blue',
input_image: PNG,
input_image_mime_type: 'image/png',
},
{
prompt: 'combine these',
input_images: [PNG, PNG],
imageSize: '1K',
resolution: '1k',
},
],
},
{
name: 'video generation',
driver: 'aiVideo',
method: 'generate',
providers: [
'byteplus-video-generation',
'gemini-video-generation',
'kling-video-generation',
'minimax-video-generation',
'together-video-generation',
],
routes: modelRoutes,
corpus: [
{ prompt: 'a red ball' },
{
prompt: 'a red ball',
seconds: 4,
resolution: '480p',
size: '1280x720',
fps: 24,
seed: 7,
negative_prompt: 'blur',
generate_audio: true,
output_format: 'mp4',
},
{
prompt: 'animate this',
reference_images: [PNG],
frame_images: [{ input_image: PNG, frame: 0 }],
last_frame: PNG,
input_reference: PNG,
metadata: { trace: 't' },
},
{
prompt: 'edit this',
input_video: 'data:video/mp4;base64,AAAA',
duration: '5',
width: 1280,
height: 720,
},
],
},
{
name: 'text to speech',
driver: 'aiTts',
method: 'synthesize',
providers: [
'aws-polly',
'elevenlabs',
'gemini',
'kling',
'openai',
'xai',
],
routes: providerRoutes,
corpus: [
{ text: 'Hello.' },
{
text: 'Hello.',
voice: 'Joanna',
engine: 'neural',
model: 'tts-1',
language: 'en-US',
response_format: 'mp3',
output_format: 'mp3_44100_128',
instructions: 'Calm.',
voice_settings: { stability: 0.5 },
},
{ text: 'Hello.', ssml: '<speak>Hello.</speak>' },
],
},
{
name: 'speech to text',
driver: 'aiSpeech2Txt',
method: 'transcribe',
providers: ['gemini', 'openai', 'xai'],
routes: providerRoutes,
corpus: [
{ file: AUDIO },
{
file: AUDIO,
language: 'en',
prompt: 'Names: Ada.',
temperature: 0,
response_format: 'verbose_json',
timestamp_granularities: ['word'],
logprobs: true,
diarize: true,
known_speaker_names: ['Ada'],
extra_body: { a: 1 },
chunking_strategy: 'auto',
format: true,
multichannel: false,
channels: 1,
audio_format: 'wav',
sample_rate: 16000,
},
],
},
{
name: 'speech to speech',
driver: 'aiSpeech2Speech',
method: 'convert',
providers: ['elevenlabs'],
routes: async () => [{ provider: 'elevenlabs' }],
corpus: [
{ audio: AUDIO },
{
audio: AUDIO,
voice: 'v',
model: 'eleven_multilingual_sts_v2',
voice_settings: { stability: 0.5 },
seed: 1,
remove_background_noise: true,
output_format: 'mp3_44100_128',
file_format: 'other',
optimize_streaming_latency: 1,
enable_logging: false,
},
],
},
{
name: 'OCR',
driver: 'aiOcr',
method: 'recognize',
providers: ['aws-textract', 'mistral'],
routes: async () => [
{ provider: 'aws-textract' },
{ provider: 'mistral' },
],
corpus: [
{ source: PDF },
{ file: PDF },
{
source: PNG,
pages: [0],
includeImageBase64: true,
imageLimit: 1,
imageMinSize: 10,
tableFormat: 'markdown',
extractHeader: true,
extractFooter: false,
documentAnnotationPrompt: 'Summarize.',
bboxAnnotationFormat: { type: 'json_schema' },
documentAnnotationFormat: { type: 'text' },
},
],
},
];
const key = { apiKey: 'fuzz-key' };
const aws = {
aws: { access_key: 'AKIAFUZZ', secret_key: 'x', region: 'us-west-2' },
};
const providers = {
openai: key,
'openai-completion': key,
gemini: key,
xai: key,
mistral: key,
'mistral-ocr': key,
minimax: key,
'together-ai': key,
byteplus: key,
elevenlabs: key,
kling: key,
'replicate-image-generation': key,
'cloudflare-image-generation': { apiToken: 'fuzz-key', accountId: 'acct' },
'aws-polly': aws,
'aws-textract': aws,
ollama: { enabled: false },
};
const refused = { status: 400, body: { error: { message: 'refused' } } };
let server: PuterServer;
// File input needs a user, so the calls run as one, not as the system actor.
let actor: Actor;
beforeAll(async () => {
upstream.install();
upstream.respond = () => refused;
const refuse = async () => {
throw Object.assign(new Error('refused'), {
name: 'ValidationException',
$metadata: { httpStatusCode: 400 },
});
};
vi.spyOn(PollyClient.prototype, 'send').mockImplementation(refuse);
vi.spyOn(TextractClient.prototype, 'send').mockImplementation(refuse);
server = await setupTestServer({
providers,
unlimitedMetering: true,
} as never);
const user = await server.stores.user.create({
username: 'media-fuzz',
uuid: '6b1f3c1e-5d0a-4f6e-9c53-0d1f6a2b7c11',
password: null,
email: '[email protected]',
requires_email_confirmation: false,
});
actor = {
user: {
id: user.id,
uuid: user.uuid,
username: user.username,
email: user.email ?? null,
email_confirmed: true,
} as Actor['user'],
};
}, 120_000);
afterAll(async () => {
await server?.shutdown();
upstream.uninstall();
vi.restoreAllMocks();
});
describe.each(TARGETS)('$name driver seeded fuzzing', (target) => {
it('rejects a malformed request with a classified error on every provider', async () => {
const driver = (server.drivers as unknown as Record<string, any>)[
target.driver
];
const routes = await target.routes(driver);
const byProvider = new Map<string, Route[]>();
for (const route of routes) {
byProvider.set(route.provider, [
...(byProvider.get(route.provider) ?? []),
route,
]);
}
expect([...byProvider.keys()].sort()).toEqual(target.providers);
const pinned = (random: Random) => {
const route: Record<string, unknown> = {
...pick(
random,
byProvider.get(pick(random, target.providers))!,
),
};
// Now and then the route itself is the malformed part.
if (random() < 0.05) {
route[pick(random, ['provider', 'model'])] =
hostileValue(random);
}
return route;
};
const findings = await fuzzCalls({
seed: 0x3ed1a,
cases: fuzzCases(1000),
corpus: target.corpus,
optionNames: target.corpus.flatMap((request) =>
Object.keys(request),
),
pinned,
call: async (args) => {
const result = await withTestActor(
() => driver[target.method](args),
actor,
);
if (isDriverStreamResult(result))
await readStreamResult(result);
},
});
findings.assertNone();
}, 600_000);
});
@@ -49,16 +49,17 @@ import {
} from '@aws-sdk/client-textract';
import { v4 as uuidv4 } from 'uuid';
import type { Actor } from '../../core/actor.js';
import { runWithContext } from '../../core/context.js';
import { PuterServer } from '../../server.js';
import type { MeteringService } from '../../services/metering/MeteringService.js';
import { buildPdf } from '../../testFixtures/pdf.js';
import { setupTestServer } from '../../testUtil.js';
import { generateDefaultFsentries } from '../../util/userProvisioning.js';
import { SDKValidationError } from '@mistralai/mistralai/models/errors/sdkvalidationerror.js';
import type { Actor } from '../../../../core/actor.js';
import { runWithContext } from '../../../../core/context.js';
import { PuterServer } from '../../../../server.js';
import type { MeteringService } from '../../../../services/metering/MeteringService.js';
import { buildPdf } from '../../../../testFixtures/pdf.js';
import { setupTestServer } from '../../../../testUtil.js';
import { generateDefaultFsentries } from '../../../../util/userProvisioning.js';
import type { OCRDriver } from './OCRDriver.js';
import { OCR_COSTS } from './costs.js';
import textractUsWest2 from './fixtures/textract-us-west-2.json';
import textractUsWest2 from '../../providers/awsTextract/ocr/fixtures/textract-us-west-2.json';
// ── SDK mocks ───────────────────────────────────────────────────────
//
@@ -300,6 +301,19 @@ describe('OCRDriver.recognize argument validation', () => {
).rejects.toMatchObject({ statusCode: 401 });
});
it('answers 400 for an option value that cannot be text', async () => {
await expect(
driver.recognize({
source: dataUrl(Buffer.from('x'), 'image/png'),
imageLimit: { toString: null },
} as never),
).rejects.toMatchObject({
statusCode: 400,
legacyCode: 'bad_request',
message: '`imageLimit` is not a usable value',
});
});
it('throws 400 when neither source nor file is provided', async () => {
const { actor } = await makeUser();
await expect(
@@ -986,6 +1000,27 @@ describe('OCRDriver.recognize (mistral)', () => {
expect(result.text).toBe('Total: 42');
});
it('answers 400 when the Mistral SDK refuses the options before sending', async () => {
const { actor } = await makeUser();
mistralOcrProcessMock.mockRejectedValueOnce(
new SDKValidationError('Input validation failed', new Error(), {}),
);
await expect(
withActor(actor, () =>
driver.recognize({
source: dataUrl(Buffer.from('x'), 'application/pdf'),
provider: 'mistral',
pages: true,
} as never),
),
).rejects.toMatchObject({
statusCode: 400,
legacyCode: 'bad_request',
});
expect(incrementUsageSpy).not.toHaveBeenCalled();
});
it('meters per-page Mistral OCR usage from costs.ts', async () => {
const { actor } = await makeUser();
mistralOcrProcessMock.mockResolvedValueOnce({
@@ -0,0 +1,262 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { Context } from '../../../../core/context.js';
import { HttpError } from '../../../../core/http/HttpError.js';
import { awsCredentialsFrom } from '../../../../util/awsCredentials.js';
import { assertActorHasCredits } from '../../../../services/metering/enforcement.js';
import { PuterDriver } from '../../../types.js';
import {
type AiMeteringService,
withAiCostFactor,
} from '../../../../util/aiCostFactor.js';
import { AI_CONCURRENT, AI_RATE_LIMIT } from '../../shared/aiLimits.js';
import { assertScalarOptions } from '../../shared/scalarOptions.js';
import { resolveFileInput } from '../../../util/fileInput.js';
import {
type ProviderCatalog,
ProviderRegistry,
} from '../../shared/providerRegistry.js';
import { readProviderKey } from '../../../../util/aiProviderKey.js';
import {
TEXTRACT_PAGE_USAGE_TYPE,
TextractOcrProvider,
} from '../../providers/awsTextract/ocr/TextractOcrProvider.js';
import { MistralOcrProvider } from '../../providers/mistral/ocr/MistralOcrProvider.js';
import { badRequest } from './common.js';
import { OCR_COSTS } from './costs.js';
import {
DEFAULT_OCR_MODEL,
findOcrModel,
OCR_MAX_INPUT_BYTES,
RETIRED_OCR_MODELS,
type OcrModel,
type OcrProviderId,
} from './models.js';
import type { IOcrProvider, RecognizeArgs } from './types.js';
// Aliases callers may use in place of a canonical provider id. Resolved here
// rather than in the SDK so a new alias reaches every caller at once.
const OCR_CATALOG: ProviderCatalog = {
label: 'OCR',
ids: ['aws-textract', 'mistral'],
defaultId: 'aws-textract',
aliases: {
aws: 'aws-textract',
'aws-textract': 'aws-textract',
textract: 'aws-textract',
mistral: 'mistral',
'mistral-ocr': 'mistral',
},
};
const NOT_CONFIGURED: Record<OcrProviderId, string> = {
'aws-textract': 'AWS credentials not configured',
mistral: 'Mistral OCR not configured',
};
// Options the providers read as text, a number or a flag.
const SCALAR_OPTIONS = [
'provider',
'model',
'documentAnnotationPrompt',
'tableFormat',
'imageLimit',
'imageMinSize',
];
/**
* Driver implementing `puter-ocr`, document OCR, with two providers:
* `aws-textract` and `mistral`.
*/
export class OCRDriver extends PuterDriver {
readonly driverInterface = 'puter-ocr';
readonly driverName = 'ai-ocr';
// Shared AI policy — see `drivers/ai/shared/aiLimits.ts` for the tier table.
readonly rateLimit = AI_RATE_LIMIT;
readonly concurrent = AI_CONCURRENT;
/** Metering scoped to this driver. */
get #aiMetering(): AiMeteringService {
return withAiCostFactor(
this.services.metering,
this.clients.event,
this.driverName,
);
}
/**
* Textract is reported at the configured region's first-tier rate. A call
* that reads its file from a bucket in another region is billed at that
* region's rate instead, which this per-usage-type report can't show.
*/
override getReportedCosts() {
return Object.entries(OCR_COSTS).map(([usageType, ucentsPerUnit]) => ({
usageType,
ucentsPerUnit:
usageType === TEXTRACT_PAGE_USAGE_TYPE
? (this.#textract?.reportedPageRate() ?? ucentsPerUnit)
: ucentsPerUnit,
unit: 'page',
source: 'driver:aiOcr',
}));
}
// Older SDK bundles name the provider in the driver slot instead of
// passing `{ provider }`; `#resolveModel` reads the requested alias
// back off the Context.
readonly driverAliases = [...OCR_CATALOG.ids];
readonly isDefault = true;
/** The registered Textract provider, whose live rate the cost report reads. */
#textract: TextractOcrProvider | undefined;
#providers = new ProviderRegistry<IOcrProvider>(OCR_CATALOG);
override onServerStart() {
const providers = this.config.providers ?? {};
const metering = this.#aiMetering;
const textract = providers['aws-textract'] as
Record<string, unknown> | undefined;
const textractAws = (textract?.aws ?? textract) as
Record<string, unknown> | undefined;
const textractCredentials = awsCredentialsFrom(textractAws);
if (textractCredentials) {
this.#textract = new TextractOcrProvider(metering, {
credentials: textractCredentials,
region:
(textractAws?.region as string | undefined) ??
(textract?.region as string | undefined),
});
this.#providers.register('aws-textract', this.#textract);
}
const mistralKey = readProviderKey(providers['mistral-ocr']);
if (mistralKey) {
try {
this.#providers.register(
'mistral',
new MistralOcrProvider(metering, { apiKey: mistralKey }),
);
} catch (e) {
console.warn(
'[OCRDriver] Failed to init Mistral:',
(e as Error).message,
);
}
}
}
async recognize(args: RecognizeArgs) {
assertScalarOptions(args, SCALAR_OPTIONS);
if (args.test_mode) return sampleResponse();
const model = this.#resolveModel(args);
const actor = Context.get('actor');
if (!actor)
throw new HttpError(401, 'Authentication required', {
legacyCode: 'unauthorized',
});
const input = args.source ?? args.file;
if (!input) throw badRequest('`source` is required');
const provider = this.#providers.get(model.provider);
if (!provider)
throw new HttpError(500, NOT_CONFIGURED[model.provider], {
legacyCode: 'internal_error',
});
// Cheap, cached check: reject a zero-balance actor before paying for
// the file load and page estimate, which a real credit check would
// reject anyway once it runs.
await assertActorHasCredits(this.services.metering, actor, this.config);
// A Puter file is resolved and access-checked here but read only if
// the provider needs its bytes; Textract reads it straight from S3.
const resolved = await resolveFileInput(
this.stores,
this.services.fs,
actor,
input,
{
acceptWebInput: true,
maxBytes: OCR_MAX_INPUT_BYTES[model.provider],
},
);
return provider.recognize(resolved, args, model, actor);
}
/**
* Decide which model handles a call. A `model` picks its own provider; an
* explicit `provider`, then the legacy driver alias the caller dispatched
* through, then whichever provider is configured pick that provider's
* default model.
*/
#resolveModel(args: RecognizeArgs): OcrModel {
const provider = args.provider
? (this.#providers.resolve(args.provider) as OcrProviderId)
: undefined;
if (args.model !== undefined) {
if (typeof args.model !== 'string' || !args.model.trim())
throw badRequest('`model` must be a non-empty string');
const retired = RETIRED_OCR_MODELS[args.model.trim().toLowerCase()];
if (retired)
throw badRequest(
`${args.model} is no longer available: ${retired}`,
);
const model = findOcrModel(args.model);
if (!model) throw badRequest(`Unknown OCR model: ${args.model}`);
if (provider && model.provider !== provider)
throw badRequest(
`OCR model ${args.model} is not served by provider ${args.provider}`,
);
return model;
}
const hinted =
provider ?? this.#providers.normalize(Context.get('driverName'));
if (!hinted && this.#providers.names().length === 0)
throw new HttpError(500, 'No OCR provider configured', {
legacyCode: 'internal_error',
});
const chosen = (hinted ?? this.#providers.defaultId()) as OcrProviderId;
return findOcrModel(DEFAULT_OCR_MODEL[chosen])!;
}
}
function sampleResponse() {
return {
model: 'test-mode',
blocks: [
{
type: 'text/puter:sample-output',
confidence: 1,
text: 'test_mode is enabled; this is a sample OCR response.',
page: 0,
},
],
text: 'test_mode is enabled; this is a sample OCR response.',
};
}
@@ -0,0 +1,103 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import type { Actor } from '../../../../core/actor.js';
import { HttpError } from '../../../../core/http/HttpError.js';
import { insufficientCreditsError } from '../../../../services/metering/enforcement.js';
import type { CreditHold } from '../../../../services/metering/types.js';
import { mimeFromName } from '../../../../util/fileSigning.js';
import type { LoadedFile, ResolvedFileInput } from '../../../util/fileInput.js';
import type { AiMeteringService } from '../../../../util/aiCostFactor.js';
import {
OCR_MAX_INPUT_BYTES,
OCR_MAX_PAGES,
type OcrProviderId,
} from './models.js';
import { countPdfPages } from './pdfPages.js';
export const badRequest = (message: string) =>
new HttpError(400, message, { legacyCode: 'bad_request' });
export const inputMimeType = (loaded: LoadedFile): string =>
loaded.mimeType ??
mimeFromName(loaded.filename) ??
'application/octet-stream';
/**
* Declared documents (PDF, DOCX, PPTX, ...) and PDF bytes are documents; images
* and untyped bytes are read as a single image.
*/
export const isDocumentInput = (loaded: LoadedFile): boolean => {
const mime = inputMimeType(loaded);
return (
loaded.buffer.subarray(0, 4).toString('latin1') === '%PDF' ||
loaded.filename.toLowerCase().endsWith('.pdf') ||
(!mime.startsWith('image/') && mime !== 'application/octet-stream')
);
};
/**
* The most pages a call can bill, known before the provider runs: a PDF's page
* count, or a size-based guess for other documents, capped by a `pages`
* selection and by what the provider reads in one call. Reads the input only
* for a provider that takes more than one page.
*/
export const estimateOcrPages = async (
input: ResolvedFileInput,
provider: OcrProviderId,
selection?: unknown,
): Promise<number> => {
const maxPages = OCR_MAX_PAGES[provider];
if (maxPages <= 1) return 1;
const loaded = await input.load();
if (!isDocumentInput(loaded)) return 1;
// A document we can't count is billed in proportion to its size, reaching
// the provider's page limit at its size limit (20 pages per MB for
// Mistral). Small files stay cheap; a large one can't pass as one page.
const documentPages =
(await countPdfPages(loaded.buffer, maxPages)) ??
Math.ceil(
(loaded.buffer.length * maxPages) / OCR_MAX_INPUT_BYTES[provider],
);
const selected = Array.isArray(selection)
? new Set(
selection.filter(
(page) => Number.isInteger(page) && (page as number) >= 0,
),
).size
: 0;
return Math.max(1, Math.min(documentPages, maxPages, selected || Infinity));
};
/**
* Refuse a call the balance can't cover, and hold its cost while the provider
* runs so the account's concurrent calls see it. Usage is still metered from
* the pages the provider reports.
*/
export const holdOcrCredits = async (
metering: AiMeteringService,
actor: Actor,
usageType: string,
cost: number,
): Promise<CreditHold> => {
// Priced at the cost factor usage is recorded at.
const hold = await metering.reserveAiCredits(actor, usageType, cost);
if (!hold) throw insufficientCreditsError();
return hold;
};
@@ -18,7 +18,7 @@
*/
import { describe, expect, it } from 'vitest';
import { buildPdf } from '../../testFixtures/pdf.js';
import { buildPdf } from '../../../../testFixtures/pdf.js';
import { countPdfPages } from './pdfPages.js';
const pdf = (body: string) => Buffer.from(`%PDF-1.7\n${body}`, 'latin1');
@@ -0,0 +1,63 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import type { Actor } from '../../../../core/actor.js';
import type { ResolvedFileInput } from '../../../util/fileInput.js';
import type { OcrModel } from './models.js';
export interface RecognizeArgs {
source?: unknown;
file?: unknown;
provider?: string;
model?: string;
// Mistral-specific options — ignored by Textract.
pages?: number[];
includeImageBase64?: boolean;
imageLimit?: number;
imageMinSize?: number;
bboxAnnotationFormat?: unknown;
documentAnnotationFormat?: unknown;
documentAnnotationPrompt?: string;
tableFormat?: string;
extractHeader?: boolean;
extractFooter?: boolean;
test_mode?: boolean;
}
/** One recognized line, in the same shape for every provider. */
export interface OcrBlock {
type: string;
text: string;
confidence?: number;
page: number;
}
/** One vendor's OCR, bound to the credentials this deployment configured. */
export interface IOcrProvider {
/**
* Recognizes one input with one of this provider's models: holds the
* estimated cost while the vendor runs, then meters the pages it reports.
*/
recognize(
input: ResolvedFileInput,
args: RecognizeArgs,
model: OcrModel,
actor: Actor,
): Promise<unknown>;
}
@@ -41,25 +41,24 @@ import {
} from 'vitest';
import { v4 as uuidv4 } from 'uuid';
import type { Actor } from '../../core/actor.js';
import { runWithContext } from '../../core/context.js';
import { PuterServer } from '../../server.js';
import type { MeteringService } from '../../services/metering/MeteringService.js';
import { setupTestServer } from '../../testUtil.js';
import { generateDefaultFsentries } from '../../util/userProvisioning.js';
import type { DriverStreamResult } from '../meta.js';
import type { Actor } from '../../../../core/actor.js';
import { runWithContext } from '../../../../core/context.js';
import { PuterServer } from '../../../../server.js';
import type { MeteringService } from '../../../../services/metering/MeteringService.js';
import { setupTestServer, withAiCostFactorHook } from '../../../../testUtil.js';
import { generateDefaultFsentries } from '../../../../util/userProvisioning.js';
import type { DriverStreamResult } from '../../../meta.js';
import {
OggOpusClock,
outputBytesPerSecond,
type VoiceChangerDriver,
wavDataOffset,
} from './VoiceChangerDriver.js';
} from '../../providers/elevenlabs/speechToSpeech/measuredOutput.js';
import type { VoiceChangerDriver } from './VoiceChangerDriver.js';
import {
callerError,
readStreamResult,
withAiCostFactorHook,
} from '../integrationTestUtil.js';
import { VOICE_CHANGER_COSTS } from './costs.js';
} from '../../shared/integrationTestUtil.js';
import { VOICE_CHANGER_COSTS } from '../../providers/elevenlabs/speechToSpeech/costs.js';
const { webFetchMock } = vi.hoisted(() => ({ webFetchMock: vi.fn() }));
vi.mock('undici', async (importOriginal) => ({
@@ -191,6 +190,36 @@ describe('VoiceChangerDriver.convert argument validation', () => {
).rejects.toMatchObject({ statusCode: 400 });
});
it('answers 400 for an option value that cannot be text', async () => {
await expect(
driver.convert({
audio: dataUrl(Buffer.from('audio'), 'audio/mpeg'),
enable_logging: { toString: null },
} as never),
).rejects.toMatchObject({
statusCode: 400,
legacyCode: 'bad_request',
message: '`enable_logging` is not a usable value',
});
expect(fetchSpy).not.toHaveBeenCalled();
});
it('rejects an output format that is not text before calling ElevenLabs', async () => {
const { actor } = await makeUser();
await expect(
withActor(actor, () =>
driver.convert({
audio: dataUrl(Buffer.from('audio'), 'audio/mpeg'),
output_format: -1,
} as never),
),
).rejects.toMatchObject({
statusCode: 400,
legacyCode: 'bad_request',
});
expect(fetchSpy).not.toHaveBeenCalled();
});
it.each([
['an unpriced', 'eleven_multilingual_sts_v1'],
['a malformed', 'eleven_english_sts_v2&x=1'],
@@ -0,0 +1,153 @@
/*
* Copyright (C) 2024-present Puter Technologies Inc.
*
* This file is part of Puter.
*
* Puter is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published
* by the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { Context } from '../../../../core/context.js';
import { HttpError } from '../../../../core/http/HttpError.js';
import type { DriverStreamResult } from '../../../meta.js';
import { ElevenLabsSpeechToSpeechProvider } from '../../providers/elevenlabs/speechToSpeech/ElevenLabsSpeechToSpeechProvider.js';
import { VOICE_CHANGER_COSTS } from '../../providers/elevenlabs/speechToSpeech/costs.js';
import { PuterDriver } from '../../../types.js';
import {
type AiMeteringService,
withAiCostFactor,
} from '../../../../util/aiCostFactor.js';
import { AI_CONCURRENT, AI_RATE_LIMIT } from '../../shared/aiLimits.js';
import { assertScalarOptions } from '../../shared/scalarOptions.js';
import {
type ProviderCatalog,
ProviderRegistry,
} from '../../shared/providerRegistry.js';
import { readProviderKey } from '../../../../util/aiProviderKey.js';
import { SAMPLE_AUDIO_URL } from '../../shared/testMode.js';
import type { ConvertArgs } from './types.js';
/**
* Driver implementing `puter-speech2speech` — voice changer. Currently a single
* provider (ElevenLabs).
*/
const VOICE_CHANGER_CATALOG: ProviderCatalog = {
label: 'Speech-to-speech',
ids: ['elevenlabs'],
defaultId: 'elevenlabs',
aliases: { elevenlabs: 'elevenlabs' },
};
// Options the providers read as text, a number or a flag.
const SCALAR_OPTIONS = [
'provider',
'voice',
'voice_id',
'voiceId',
'model',
'model_id',
'seed',
'output_format',
'file_format',
'optimize_streaming_latency',
'enable_logging',
];
export class VoiceChangerDriver extends PuterDriver {
readonly driverInterface = 'puter-speech2speech';
readonly driverName = 'ai-speech2speech';
// Older SDK bundles name the provider in the driver slot.
readonly driverAliases = ['elevenlabs-voice-changer'];
readonly isDefault = true;
// Shared AI policy — see `drivers/ai/shared/aiLimits.ts` for the tier table.
readonly rateLimit = AI_RATE_LIMIT;
readonly concurrent = AI_CONCURRENT;
/** Metering scoped to this driver. */
get #aiMetering(): AiMeteringService {
return withAiCostFactor(
this.services.metering,
this.clients.event,
this.driverName,
);
}
override getReportedCosts(): Record<string, unknown>[] {
return Object.entries(VOICE_CHANGER_COSTS).map(
([usageType, ucentsPerUnit]) => ({
usageType,
ucentsPerUnit,
unit: 'second',
source: 'driver:aiSpeech2Speech',
}),
);
}
#providers = new ProviderRegistry<ElevenLabsSpeechToSpeechProvider>(
VOICE_CHANGER_CATALOG,
);
override onServerStart() {
const elevenlabs = this.config.providers?.elevenlabs;
const apiKey = readProviderKey(elevenlabs);
if (!apiKey) return;
this.#providers.register(
'elevenlabs',
new ElevenLabsSpeechToSpeechProvider(
{
stores: this.stores,
fs: this.services.fs,
metering: this.#aiMetering,
},
{
apiKey,
apiBaseUrl: elevenlabs?.apiBaseUrl as string | undefined,
defaultVoiceId: elevenlabs?.defaultVoiceId as
string | undefined,
defaultModelId: elevenlabs?.speechToSpeechModelId as
string | undefined,
},
),
);
}
async convert(
args: ConvertArgs,
): Promise<DriverStreamResult | { url: string; content_type: string }> {
assertScalarOptions(args, SCALAR_OPTIONS);
// Only one provider exists today, but naming a different one should
// fail loudly rather than quietly convert with this one.
const providerName = this.#providers.resolve(args.provider);
if (args.test_mode) {
return { url: SAMPLE_AUDIO_URL, content_type: 'audio/mpeg' };
}
const elevenlabs = this.#providers.get(providerName);
if (!elevenlabs) {
throw new HttpError(500, 'ElevenLabs API key not configured', {
legacyCode: 'internal_error',
});
}
const actor = Context.get('actor');
if (!actor)
throw new HttpError(401, 'Authentication required', {
legacyCode: 'unauthorized',
});
return elevenlabs.convert(args, actor);
}
}
@@ -17,17 +17,23 @@
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
/**
* Base class for image generation providers.
*
* Concrete providers (OpenAI, Gemini, etc.) extend this and implement
* `generate`, `models`, and `getDefaultModel`.
*/
/** Types for the `puter-speech2speech` driver interface. */
import type { IImageProvider, IImageModel, IGenerateParams } from '../types.js';
export abstract class ImageProvider implements IImageProvider {
abstract generate(params: IGenerateParams): Promise<string>;
abstract models(): IImageModel[] | Promise<IImageModel[]>;
abstract getDefaultModel(): string;
export interface ConvertArgs {
audio: unknown;
provider?: string;
voice?: string;
voice_id?: string;
voiceId?: string;
model?: string;
model_id?: string;
voice_settings?: unknown;
voiceSettings?: unknown;
seed?: number;
remove_background_noise?: boolean;
output_format?: string;
file_format?: string;
optimize_streaming_latency?: number;
enable_logging?: boolean;
test_mode?: boolean;
}
@@ -43,15 +43,14 @@ import {
} from 'vitest';
import { v4 as uuidv4 } from 'uuid';
import type { Actor } from '../../core/actor.js';
import { runWithContext } from '../../core/context.js';
import { PuterServer } from '../../server.js';
import type { MeteringService } from '../../services/metering/MeteringService.js';
import { setupTestServer } from '../../testUtil.js';
import { generateDefaultFsentries } from '../../util/userProvisioning.js';
import { withAiCostFactorHook } from '../integrationTestUtil.js';
import type { Actor } from '../../../../core/actor.js';
import { runWithContext } from '../../../../core/context.js';
import { PuterServer } from '../../../../server.js';
import type { MeteringService } from '../../../../services/metering/MeteringService.js';
import { setupTestServer, withAiCostFactorHook } from '../../../../testUtil.js';
import { generateDefaultFsentries } from '../../../../util/userProvisioning.js';
import type { SpeechToTextDriver } from './SpeechToTextDriver.js';
import { SPEECH_TO_TEXT_COSTS } from './costs.js';
import { SPEECH_TO_TEXT_COSTS } from '../../providers/openai/speechToText/costs.js';
// ── OpenAI SDK mock ─────────────────────────────────────────────────
//
@@ -270,6 +269,23 @@ describe('SpeechToTextDriver.transcribe test_mode', () => {
// ── Argument validation ─────────────────────────────────────────────
describe('SpeechToTextDriver argument validation', () => {
it.each(['transcribe', 'translate'] as const)(
'answers 400 from %s for an option value that cannot be text',
async (method) => {
await expect(
driver[method]({
file: dataUrl(Buffer.from('a'), 'audio/mp3'),
language: { toString: null },
} as never),
).rejects.toMatchObject({
statusCode: 400,
legacyCode: 'bad_request',
message: '`language` is not a usable value',
});
expect(transcriptionsCreateMock).not.toHaveBeenCalled();
},
);
it('rejects streaming with 400 — not yet supported', async () => {
const { actor } = await makeUser();
await expect(
@@ -17,22 +17,24 @@
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { Context } from '../../core/context.js';
import { HttpError } from '../../core/http/HttpError.js';
import { PuterDriver } from '../types.js';
import { Context } from '../../../../core/context.js';
import { HttpError } from '../../../../core/http/HttpError.js';
import { PuterDriver } from '../../../types.js';
import {
type AiMeteringService,
withAiCostFactor,
} from '../util/aiCostFactor.js';
import { AI_CONCURRENT, AI_RATE_LIMIT } from '../util/aiLimits.js';
import { ProviderRegistry, readProviderKey } from '../util/providerRegistry.js';
} from '../../../../util/aiCostFactor.js';
import { AI_CONCURRENT, AI_RATE_LIMIT } from '../../shared/aiLimits.js';
import { assertScalarOptions } from '../../shared/scalarOptions.js';
import { ProviderRegistry } from '../../shared/providerRegistry.js';
import { readProviderKey } from '../../../../util/aiProviderKey.js';
import {
SPEECH_TO_TEXT_CATALOG,
SPEECH_TO_TEXT_DRIVER_ALIASES,
} from './providerAliases.js';
import { GeminiSpeechToTextProvider } from './providers/gemini/GeminiSpeechToTextProvider.js';
import { OpenAISpeechToTextProvider } from './providers/openai/OpenAISpeechToTextProvider.js';
import { XAISpeechToTextProvider } from './providers/xai/XAISpeechToTextProvider.js';
import { GeminiSpeechToTextProvider } from '../../providers/gemini/speechToText/GeminiSpeechToTextProvider.js';
import { OpenAISpeechToTextProvider } from '../../providers/openai/speechToText/OpenAISpeechToTextProvider.js';
import { XAISpeechToTextProvider } from '../../providers/xai/speechToText/XAISpeechToTextProvider.js';
import type {
ISpeechToTextDeps,
ISpeechToTextModel,
@@ -40,6 +42,20 @@ import type {
ITranscribeArgs,
} from './types.js';
// Options the providers read as text, a number or a flag.
const SCALAR_OPTIONS = [
'provider',
'model',
'response_format',
'language',
'prompt',
'temperature',
'format',
'channels',
'audio_format',
'sample_rate',
];
/**
* Driver implementing the `puter-speech2txt` interface.
*
@@ -56,7 +72,7 @@ export class SpeechToTextDriver extends PuterDriver {
readonly driverAliases = [...SPEECH_TO_TEXT_DRIVER_ALIASES];
readonly isDefault = true;
// Shared AI policy — see `drivers/util/aiLimits.ts` for the tier table.
// Shared AI policy — see `drivers/ai/shared/aiLimits.ts` for the tier table.
// One bucket covers every provider, keyed by interface+method+user.
readonly rateLimit = AI_RATE_LIMIT;
readonly concurrent = AI_CONCURRENT;
@@ -111,10 +127,12 @@ export class SpeechToTextDriver extends PuterDriver {
}
async transcribe(args: ITranscribeArgs) {
assertScalarOptions(args, SCALAR_OPTIONS);
return this.#provider(args).transcribe(this.#providerArgs(args));
}
async translate(args: ITranscribeArgs) {
assertScalarOptions(args, SCALAR_OPTIONS);
return this.#provider(args).translate(this.#providerArgs(args));
}
@@ -22,15 +22,15 @@
* upstream API and exposes the unified `ISpeechToTextProvider` contract.
*/
import { Context } from '../../../core/context.js';
import { HttpError } from '../../../core/http/HttpError.js';
import type { Actor } from '../../../core/actor.js';
import { Context } from '../../../../core/context.js';
import { HttpError } from '../../../../core/http/HttpError.js';
import type { Actor } from '../../../../core/actor.js';
import type {
ISpeechToTextDeps,
ISpeechToTextModel,
ISpeechToTextProvider,
ITranscribeArgs,
} from '../types.js';
} from './types.js';
export abstract class SpeechToTextProvider implements ISpeechToTextProvider {
abstract readonly providerName: string;
@@ -17,7 +17,7 @@
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import type { ProviderCatalog } from '../util/providerRegistry.js';
import type { ProviderCatalog } from '../../shared/providerRegistry.js';
/**
* Canonical `puter-speech2txt` provider ids and the aliases callers may use in
@@ -19,8 +19,8 @@
/** Types for the `puter-speech2txt` driver interface. */
import type { AiMeteringService } from '../util/aiCostFactor.js';
import type { loadFileInput } from '../util/fileInput.js';
import type { AiMeteringService } from '../../../../util/aiCostFactor.js';
import type { loadFileInput } from '../../../util/fileInput.js';
/** The layers a provider needs to read its audio input and meter usage. */
export interface ISpeechToTextDeps {
@@ -40,16 +40,13 @@ import {
type MockInstance,
} from 'vitest';
import { runWithContext } from '../../core/context.js';
import { SYSTEM_ACTOR } from '../../core/actor.js';
import { PuterServer } from '../../server.js';
import { setupTestServer } from '../../testUtil.js';
import {
readStreamResult,
withAiCostFactorHook,
} from '../integrationTestUtil.js';
import { runWithContext } from '../../../../core/context.js';
import { SYSTEM_ACTOR } from '../../../../core/actor.js';
import { PuterServer } from '../../../../server.js';
import { setupTestServer, withAiCostFactorHook } from '../../../../testUtil.js';
import { readStreamResult } from '../../shared/integrationTestUtil.js';
import type { TTSDriver } from './TTSDriver.js';
import pollyPriceList from './providers/awsPolly/fixtures/us-west-2.json';
import pollyPriceList from '../../providers/awsPolly/textToSpeech/fixtures/us-west-2.json';
// ── SDK mocks ──────────────────────────────────────────────────────
//
@@ -221,6 +218,19 @@ describe('TTSDriver.synthesize authentication', () => {
driver.synthesize({ text: 'hi' } as never),
).rejects.toMatchObject({ statusCode: 401 });
});
it('answers 400 for an option value that cannot be text', async () => {
await expect(
driver.synthesize({
text: 'hi',
language: { toString: null },
} as never),
).rejects.toMatchObject({
statusCode: 400,
legacyCode: 'bad_request',
message: '`language` is not a usable value',
});
});
});
// ── Provider routing ────────────────────────────────────────────────
@@ -17,26 +17,28 @@
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { Context } from '../../core/context.js';
import { HttpError } from '../../core/http/HttpError.js';
import { awsCredentialsFrom } from '../../util/awsCredentials.js';
import type { DriverStreamResult } from '../meta.js';
import { PuterDriver } from '../types.js';
import { Context } from '../../../../core/context.js';
import { HttpError } from '../../../../core/http/HttpError.js';
import { awsCredentialsFrom } from '../../../../util/awsCredentials.js';
import type { DriverStreamResult } from '../../../meta.js';
import { PuterDriver } from '../../../types.js';
import {
type AiMeteringService,
withAiCostFactor,
} from '../util/aiCostFactor.js';
import { AI_CONCURRENT, AI_RATE_LIMIT } from '../util/aiLimits.js';
import { usesGoogleGeminiEndpoint } from '../util/geminiRest.js';
import { ProviderRegistry, readProviderKey } from '../util/providerRegistry.js';
} from '../../../../util/aiCostFactor.js';
import { AI_CONCURRENT, AI_RATE_LIMIT } from '../../shared/aiLimits.js';
import { assertScalarOptions } from '../../shared/scalarOptions.js';
import { usesGoogleGeminiEndpoint } from '../../providers/gemini/geminiRest.js';
import { ProviderRegistry } from '../../shared/providerRegistry.js';
import { readProviderKey } from '../../../../util/aiProviderKey.js';
import { TTS_CATALOG, TTS_DRIVER_ALIASES } from './providerAliases.js';
import { AWSPollyTTSProvider } from './providers/awsPolly/AWSPollyTTSProvider.js';
import { ElevenLabsTTSProvider } from './providers/elevenlabs/ElevenLabsTTSProvider.js';
import { GeminiTTSProvider } from './providers/gemini/GeminiTTSProvider.js';
import { KlingTTSProvider } from './providers/kling/KlingTTSProvider.js';
import { OpenAITTSProvider } from './providers/openai/OpenAITTSProvider.js';
import { SpeechifyTTSProvider } from './providers/speechify/SpeechifyTTSProvider.js';
import { XAITTSProvider } from './providers/xai/XAITTSProvider.js';
import { AWSPollyTTSProvider } from '../../providers/awsPolly/textToSpeech/AWSPollyTTSProvider.js';
import { ElevenLabsTTSProvider } from '../../providers/elevenlabs/textToSpeech/ElevenLabsTTSProvider.js';
import { GeminiTTSProvider } from '../../providers/gemini/textToSpeech/GeminiTTSProvider.js';
import { KlingTTSProvider } from '../../providers/kling/textToSpeech/KlingTTSProvider.js';
import { OpenAITTSProvider } from '../../providers/openai/textToSpeech/OpenAITTSProvider.js';
import { SpeechifyTTSProvider } from '../../providers/speechify/textToSpeech/SpeechifyTTSProvider.js';
import { XAITTSProvider } from '../../providers/xai/textToSpeech/XAITTSProvider.js';
import type {
ISynthesizeArgs,
ITTSEngine,
@@ -44,6 +46,20 @@ import type {
ITTSVoice,
} from './types.js';
// Options the providers read as text, a number or a flag.
const SCALAR_OPTIONS = [
'text',
'voice',
'model',
'response_format',
'output_format',
'instructions',
'ssml',
'language',
'engine',
'provider',
];
/**
* Driver implementing the `puter-tts` interface.
*
@@ -63,7 +79,7 @@ export class TTSDriver extends PuterDriver {
readonly driverAliases = [...TTS_DRIVER_ALIASES];
readonly isDefault = true;
// Shared AI policy — see `drivers/util/aiLimits.ts` for the tier table.
// Shared AI policy — see `drivers/ai/shared/aiLimits.ts` for the tier table.
readonly rateLimit = AI_RATE_LIMIT;
readonly concurrent = AI_CONCURRENT;
@@ -132,6 +148,7 @@ export class TTSDriver extends PuterDriver {
async synthesize(
args: ISynthesizeArgs,
): Promise<DriverStreamResult | { url: string; content_type: string }> {
assertScalarOptions(args, SCALAR_OPTIONS);
const actor = Context.get('actor');
if (!actor)
throw new HttpError(401, 'Authentication required', {
@@ -27,11 +27,11 @@ import {
it,
vi,
} from 'vitest';
import { makeActor } from '../../../core/actor.js';
import type { DriverStreamResult } from '../../meta.js';
import { PuterServer } from '../../../server.js';
import { setupTestServer } from '../../../testUtil.js';
import { withAiCostFactor } from '../../util/aiCostFactor.js';
import { makeActor } from '../../../../core/actor.js';
import type { DriverStreamResult } from '../../../meta.js';
import { PuterServer } from '../../../../server.js';
import { setupTestServer } from '../../../../testUtil.js';
import { withAiCostFactor } from '../../../../util/aiCostFactor.js';
import {
assertMaxChars,
characterCostReport,
@@ -17,13 +17,11 @@
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import type { Actor } from '../../../core/actor.js';
import { HttpError } from '../../../core/http/HttpError.js';
import {
type AiMeteringService,
withStreamedAiCharge,
} from '../../util/aiCostFactor.js';
import type { DriverStreamResult } from '../../meta.js';
import type { Actor } from '../../../../core/actor.js';
import { HttpError } from '../../../../core/http/HttpError.js';
import { type AiMeteringService } from '../../../../util/aiCostFactor.js';
import { withStreamedAiCharge } from '../../shared/streamedAiCharge.js';
import type { DriverStreamResult } from '../../../meta.js';
/**
* Bounds a synthesis call: the whole reply, or until headers when the audio is
@@ -17,7 +17,7 @@
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import type { ProviderCatalog } from '../util/providerRegistry.js';
import type { ProviderCatalog } from '../../shared/providerRegistry.js';
/**
* Canonical `puter-tts` provider ids and the aliases callers may use in their
@@ -41,15 +41,14 @@ import {
} from 'vitest';
import { v4 as uuidv4 } from 'uuid';
import { Context, runWithContext } from '../../core/context.js';
import { HttpError } from '../../core/http/HttpError.js';
import { type Actor, SYSTEM_ACTOR } from '../../core/actor.js';
import { PuterServer } from '../../server.js';
import type { MeteringService } from '../../services/metering/MeteringService.js';
import { setupTestServer } from '../../testUtil.js';
import { withAiCostFactorHook } from '../integrationTestUtil.js';
import { generateDefaultFsentries } from '../../util/userProvisioning.js';
import { TOGETHER_VIDEO_GENERATION_MODELS } from './providers/together/models.js';
import { Context, runWithContext } from '../../../../core/context.js';
import { HttpError } from '../../../../core/http/HttpError.js';
import { type Actor, SYSTEM_ACTOR } from '../../../../core/actor.js';
import { PuterServer } from '../../../../server.js';
import type { MeteringService } from '../../../../services/metering/MeteringService.js';
import { setupTestServer, withAiCostFactorHook } from '../../../../testUtil.js';
import { generateDefaultFsentries } from '../../../../util/userProvisioning.js';
import { TOGETHER_VIDEO_GENERATION_MODELS } from '../../providers/together/video/models.js';
import type { VideoGenerationDriver } from './VideoGenerationDriver.js';
const DEFAULT_MODEL = 'togetherai:wan-ai/wan2.7-t2v';
@@ -85,8 +84,8 @@ vi.mock('together-ai', () => {
const { secureFetchMock } = vi.hoisted(() => ({ secureFetchMock: vi.fn() }));
vi.mock('../../util/secureHttp.js', async (importOriginal) => ({
...(await importOriginal<typeof import('../../util/secureHttp.js')>()),
vi.mock('../../../../util/secureHttp.js', async (importOriginal) => ({
...(await importOriginal<typeof import('../../../../util/secureHttp.js')>()),
secureFetch: secureFetchMock,
}));
@@ -413,6 +412,48 @@ describe('VideoGenerationDriver.generate provider routing', () => {
},
);
it.each([5, true, { toString: null }, ['wan']])(
'rejects the model %j, which is not a name',
async (model) => {
await expect(
withDriverName('ai-video', () =>
driver.generate({ prompt: 'hi', model } as never),
),
).rejects.toMatchObject({
statusCode: 400,
legacyCode: 'bad_request',
});
expect(togetherVideosCreateMock).not.toHaveBeenCalled();
},
);
it('answers 400 for an option value that cannot be text', async () => {
await expect(
withActor(() =>
driver.generate({
prompt: 'hi',
seed: { toString: null },
} as never),
),
).rejects.toMatchObject({
statusCode: 400,
legacyCode: 'bad_request',
message: '`seed` is not a usable value',
});
expect(togetherVideosCreateMock).not.toHaveBeenCalled();
});
it('falls back to the model default for a size that is an object', async () => {
completeTogetherJob();
await withActor(() =>
driver.generate({
prompt: 'hi',
size: { width: 1280, height: 720 },
} as never),
);
expect(togetherVideosCreateMock).toHaveBeenCalledTimes(1);
});
it.each(['__proto__', 'constructor'])(
'uses the default for an unknown provider named %s',
async (provider) => {
@@ -741,7 +782,7 @@ describe('VideoGenerationDriver metering propagation', () => {
// ── puter_output_path ─────────────────────────────────────────────
describe('VideoGenerationDriver.generate puter_output_path', () => {
const TEST_ACTOR: import('../../core/actor.js').Actor = {
const TEST_ACTOR: import('../../../../core/actor.js').Actor = {
user: { uuid: 'a1b2c3d4-e5f6-4a7b-8c9d-0e1f2a3b4c5d', id: 42, username: 'testuser' },
};
@@ -1005,7 +1046,7 @@ describe('VideoGenerationDriver.generate puter_output_path', () => {
});
it('throws 400 when actor has no user ID but puter_output_path is set', async () => {
const noIdActor: import('../../core/actor.js').Actor = {
const noIdActor: import('../../../../core/actor.js').Actor = {
user: { uuid: 'f0e1d2c3-b4a5-4968-8777-0a1b2c3d4e5f', username: 'noone' },
};
await expect(
@@ -17,32 +17,33 @@
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/
import { Context } from '../../core/context.js';
import { HttpError } from '../../core/http/HttpError.js';
import type { Actor } from '../../core/actor.js';
import { PuterDriver } from '../types.js';
import { Context } from '../../../../core/context.js';
import { HttpError } from '../../../../core/http/HttpError.js';
import type { Actor } from '../../../../core/actor.js';
import { PuterDriver } from '../../../types.js';
import {
type AiMeteringService,
withAiCostFactor,
} from '../util/aiCostFactor.js';
import { AI_CONCURRENT, AI_RATE_LIMIT } from '../util/aiLimits.js';
} from '../../../../util/aiCostFactor.js';
import { AI_CONCURRENT, AI_RATE_LIMIT } from '../../shared/aiLimits.js';
import { assertScalarOptions } from '../../shared/scalarOptions.js';
import {
resolveOutputPath,
saveGeneratedMediaToFS,
} from '../util/generatedMedia.js';
import { usesGoogleGeminiEndpoint } from '../util/geminiRest.js';
} from '../../shared/generatedMedia.js';
import { usesGoogleGeminiEndpoint } from '../../providers/gemini/geminiRest.js';
import {
dataUriByteLength,
MAX_INLINE_VIDEO_BYTES,
readPuterImageInputs,
} from '../util/mediaInput.js';
import { ModelCatalog } from '../util/modelCatalog.js';
import { readProviderKey } from '../util/providerRegistry.js';
import { BytePlusVideoProvider } from './providers/byteplus/BytePlusVideoProvider.js';
import { GeminiVideoProvider } from './providers/gemini/GeminiVideoProvider.js';
import { KlingVideoProvider } from './providers/kling/KlingVideoProvider.js';
import { MiniMaxVideoProvider } from './providers/minimax/MiniMaxVideoProvider.js';
import { TogetherVideoProvider } from './providers/together/TogetherVideoProvider.js';
} from '../../shared/mediaInput.js';
import { ModelCatalog } from '../../shared/modelCatalog.js';
import { readProviderKey } from '../../../../util/aiProviderKey.js';
import { BytePlusVideoProvider } from '../../providers/byteplus/video/BytePlusVideoProvider.js';
import { GeminiVideoProvider } from '../../providers/gemini/video/GeminiVideoProvider.js';
import { KlingVideoProvider } from '../../providers/kling/video/KlingVideoProvider.js';
import { MiniMaxVideoProvider } from '../../providers/minimax/video/MiniMaxVideoProvider.js';
import { TogetherVideoProvider } from '../../providers/together/video/TogetherVideoProvider.js';
import type {
IGenerateVideoParams,
IVideoModel,
@@ -81,6 +82,28 @@ const tierForPixels = (
return '4k';
};
// Options the providers read as text, a number or a flag.
const SCALAR_OPTIONS = [
'prompt',
'model',
'provider',
'seconds',
'duration',
'size',
'resolution',
'width',
'height',
'fps',
'steps',
'guidance_scale',
'seed',
'output_format',
'output_quality',
'negative_prompt',
'last_frame',
'puter_output_path',
];
/**
* Driver implementing the `puter-video-generation` interface.
*
@@ -106,7 +129,7 @@ export class VideoGenerationDriver extends PuterDriver {
];
readonly isDefault = true;
// Shared AI policy — see `drivers/util/aiLimits.ts` for the tier table.
// Shared AI policy — see `drivers/ai/shared/aiLimits.ts` for the tier table.
readonly rateLimit = AI_RATE_LIMIT;
readonly concurrent = AI_CONCURRENT;
@@ -164,6 +187,7 @@ export class VideoGenerationDriver extends PuterDriver {
}
async generate(args: IGenerateVideoParams) {
assertScalarOptions(args, SCALAR_OPTIONS);
const actor = Context.get('actor') as Actor | undefined;
if (!actor)
throw new HttpError(401, 'Authentication required', {
@@ -179,7 +203,7 @@ export class VideoGenerationDriver extends PuterDriver {
? await resolveOutputPath(this.services, actor, puterOutputPath)
: undefined;
if (request.model) {
if (typeof request.model === 'string') {
request.model = request.model.trim().toLowerCase();
}
@@ -595,7 +619,10 @@ export class VideoGenerationDriver extends PuterDriver {
}
#resolveModel(modelId: string, provider?: string): IVideoModel | null {
const models = this.#modelIdMap[modelId?.trim().toLowerCase()];
const models =
typeof modelId === 'string'
? this.#modelIdMap[modelId.trim().toLowerCase()]
: undefined;
if (!models || models.length === 0) return null;
if (!provider) return models[0];
@@ -21,7 +21,7 @@ import type {
IVideoModel,
IVideoProvider,
IGenerateVideoParams,
} from '../types.js';
} from './types.js';
/**
* Abstract base for AI video providers. Each provider wraps a single upstream
Loaded 100 of 401 files, more files were not shown because too many files have changed in this diff. Show more