diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 44282b5d2..67002f3b2 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -120,46 +120,50 @@ Flowise has 3 different modules in a single mono repository. Flowise support different environment variables to configure your instance. You can specify the following variables in the `.env` file inside `packages/server` folder. Read [more](https://docs.flowiseai.com/environment-variables) -| Variable | Description | Type | Default | -| ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ | ----------------------------------- | -| PORT | The HTTP port Flowise runs on | Number | 3000 | -| CORS_ORIGINS | The allowed origins for all cross-origin HTTP calls | String | | -| IFRAME_ORIGINS | The allowed origins for iframe src embedding | String | | -| FLOWISE_FILE_SIZE_LIMIT | Upload File Size Limit | String | 50mb | -| DEBUG | Print logs from components | Boolean | | -| LOG_PATH | Location where log files are stored | String | `your-path/Flowise/logs` | -| LOG_LEVEL | Different levels of logs | Enum String: `error`, `info`, `verbose`, `debug` | `info` | -| LOG_JSON_SPACES | Spaces to beautify JSON logs | | 2 | -| TOOL_FUNCTION_BUILTIN_DEP | NodeJS built-in modules to be used for Custom Tool or Function | String | | -| TOOL_FUNCTION_EXTERNAL_DEP | External modules to be used for Custom Tool or Function | String | | -| ALLOW_BUILTIN_DEP | Allow project dependencies to be used for Custom Tool or Function | Boolean | false | -| DATABASE_TYPE | Type of database to store the flowise data | Enum String: `sqlite`, `mysql`, `postgres` | `sqlite` | -| DATABASE_PATH | Location where database is saved (When DATABASE_TYPE is sqlite) | String | `your-home-dir/.flowise` | -| DATABASE_HOST | Host URL or IP address (When DATABASE_TYPE is not sqlite) | String | | -| DATABASE_PORT | Database port (When DATABASE_TYPE is not sqlite) | String | | -| DATABASE_USER | Database username (When DATABASE_TYPE is not sqlite) | String | | -| DATABASE_PASSWORD | Database password (When DATABASE_TYPE is not sqlite) | String | | -| DATABASE_NAME | Database name (When DATABASE_TYPE is not sqlite) | String | | -| DATABASE_SSL_KEY_BASE64 | Database SSL client cert in base64 (takes priority over DATABASE_SSL) | Boolean | false | -| DATABASE_SSL | Database connection overssl (When DATABASE_TYPE is postgre) | Boolean | false | -| SECRETKEY_PATH | Location where encryption key (used to encrypt/decrypt credentials) is saved | String | `your-path/Flowise/packages/server` | -| FLOWISE_SECRETKEY_OVERWRITE | Encryption key to be used instead of the key stored in SECRETKEY_PATH | String | | -| MODEL_LIST_CONFIG_JSON | File path to load list of models from your local config file | String | `/your_model_list_config_file_path` | -| STORAGE_TYPE | Type of storage for uploaded files. default is `local` | Enum String: `s3`, `local`, `gcs` | `local` | -| BLOB_STORAGE_PATH | Local folder path where uploaded files are stored when `STORAGE_TYPE` is `local` | String | `your-home-dir/.flowise/storage` | -| S3_STORAGE_BUCKET_NAME | Bucket name to hold the uploaded files when `STORAGE_TYPE` is `s3` | String | | -| S3_STORAGE_ACCESS_KEY_ID | AWS Access Key | String | | -| S3_STORAGE_SECRET_ACCESS_KEY | AWS Secret Key | String | | -| S3_STORAGE_REGION | Region for S3 bucket | String | | -| S3_ENDPOINT_URL | Custom Endpoint for S3 | String | | -| S3_FORCE_PATH_STYLE | Set this to true to force the request to use path-style addressing | Boolean | false | -| GOOGLE_CLOUD_STORAGE_PROJ_ID | The GCP project id for cloud storage & logging when `STORAGE_TYPE` is `gcs` | String | | -| GOOGLE_CLOUD_STORAGE_CREDENTIAL | The credential key file path when `STORAGE_TYPE` is `gcs` | String | | -| GOOGLE_CLOUD_STORAGE_BUCKET_NAME | Bucket name to hold the uploaded files when `STORAGE_TYPE` is `gcs` | String | | -| GOOGLE_CLOUD_UNIFORM_BUCKET_ACCESS | Enable uniform bucket level access when `STORAGE_TYPE` is `gcs` | Boolean | true | -| SHOW_COMMUNITY_NODES | Show nodes created by community | Boolean | | -| DISABLED_NODES | Hide nodes from UI (comma separated list of node names) | String | | -| TRUST_PROXY | Configure proxy trust settings for proper IP detection. Values: 'true' (trust all), 'false' (disable), number (hop count), or Express proxy values (e.g., 'loopback', 'linklocal', 'uniquelocal', IP addresses). [Learn More](https://expressjs.com/en/guide/behind-proxies.html) | Boolean/String/Number | true | +| Variable | Description | Type | Default | +| ------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ | ----------------------------------- | +| PORT | The HTTP port Flowise runs on | Number | 3000 | +| CORS_ORIGINS | The allowed origins for all cross-origin HTTP calls | String | | +| IFRAME_ORIGINS | The allowed origins for iframe src embedding | String | | +| FLOWISE_FILE_SIZE_LIMIT | Upload File Size Limit | String | 50mb | +| DEBUG | Print logs from components | Boolean | | +| LOG_PATH | Location where log files are stored | String | `your-path/Flowise/logs` | +| LOG_LEVEL | Different levels of logs | Enum String: `error`, `info`, `verbose`, `debug` | `info` | +| LOG_JSON_SPACES | Spaces to beautify JSON logs | | 2 | +| TOOL_FUNCTION_BUILTIN_DEP | NodeJS built-in modules to be used for Custom Tool or Function | String | | +| TOOL_FUNCTION_EXTERNAL_DEP | External modules to be used for Custom Tool or Function | String | | +| ALLOW_BUILTIN_DEP | Allow project dependencies to be used for Custom Tool or Function | Boolean | false | +| DATABASE_TYPE | Type of database to store the flowise data | Enum String: `sqlite`, `mysql`, `postgres` | `sqlite` | +| DATABASE_PATH | Location where database is saved (When DATABASE_TYPE is sqlite) | String | `your-home-dir/.flowise` | +| DATABASE_HOST | Host URL or IP address (When DATABASE_TYPE is not sqlite) | String | | +| DATABASE_PORT | Database port (When DATABASE_TYPE is not sqlite) | String | | +| DATABASE_USER | Database username (When DATABASE_TYPE is not sqlite) | String | | +| DATABASE_PASSWORD | Database password (When DATABASE_TYPE is not sqlite) | String | | +| DATABASE_NAME | Database name (When DATABASE_TYPE is not sqlite) | String | | +| DATABASE_SSL_KEY_BASE64 | Database SSL client cert in base64 (takes priority over DATABASE_SSL) | Boolean | false | +| DATABASE_SSL | Database connection overssl (When DATABASE_TYPE is postgre) | Boolean | false | +| SECRETKEY_PATH | Location where encryption key (used to encrypt/decrypt credentials) is saved | String | `your-path/Flowise/packages/server` | +| FLOWISE_SECRETKEY_OVERWRITE | Encryption key to be used instead of the key stored in SECRETKEY_PATH | String | | +| MODEL_LIST_CONFIG_JSON | File path to load list of models from your local config file | String | `/your_model_list_config_file_path` | +| STORAGE_TYPE | Type of storage for uploaded files. default is `local` | Enum String: `s3`, `local`, `gcs` ,`azure` | `local` | +| BLOB_STORAGE_PATH | Local folder path where uploaded files are stored when `STORAGE_TYPE` is `local` | String | `your-home-dir/.flowise/storage` | +| S3_STORAGE_BUCKET_NAME | Bucket name to hold the uploaded files when `STORAGE_TYPE` is `s3` | String | | +| S3_STORAGE_ACCESS_KEY_ID | AWS Access Key | String | | +| S3_STORAGE_SECRET_ACCESS_KEY | AWS Secret Key | String | | +| S3_STORAGE_REGION | Region for S3 bucket | String | | +| S3_ENDPOINT_URL | Custom Endpoint for S3 | String | | +| S3_FORCE_PATH_STYLE | Set this to true to force the request to use path-style addressing | Boolean | false | +| GOOGLE_CLOUD_STORAGE_PROJ_ID | The GCP project id for cloud storage & logging when `STORAGE_TYPE` is `gcs` | String | | +| GOOGLE_CLOUD_STORAGE_CREDENTIAL | The credential key file path when `STORAGE_TYPE` is `gcs` | String | | +| GOOGLE_CLOUD_STORAGE_BUCKET_NAME | Bucket name to hold the uploaded files when `STORAGE_TYPE` is `gcs` | String | | +| GOOGLE_CLOUD_UNIFORM_BUCKET_ACCESS | Enable uniform bucket level access when `STORAGE_TYPE` is `gcs` | Boolean | true | +| AZURE_BLOB_STORAGE_CONNECTION_STRING | Azure Blob Storage connection string when `STORAGE_TYPE` is `azure`. Either this or account name + key is required | String | | +| AZURE_BLOB_STORAGE_ACCOUNT_NAME | Azure storage account name when `STORAGE_TYPE` is `azure`. Required if connection string is not provided | String | | +| AZURE_BLOB_STORAGE_ACCOUNT_KEY | Azure storage account key when `STORAGE_TYPE` is `azure`. Required if connection string is not provided | String | | +| AZURE_BLOB_STORAGE_CONTAINER_NAME | Container name to hold the uploaded files when `STORAGE_TYPE` is `azure` | String | | +| SHOW_COMMUNITY_NODES | Show nodes created by community | Boolean | | +| DISABLED_NODES | Hide nodes from UI (comma separated list of node names) | String | | +| TRUST_PROXY | Configure proxy trust settings for proper IP detection. Values: 'true' (trust all), 'false' (disable), number (hop count), or Express proxy values (e.g., 'loopback', 'linklocal', 'uniquelocal', IP addresses). [Learn More](https://expressjs.com/en/guide/behind-proxies.html) | Boolean/String/Number | true | You can also specify the env variables when using `npx`. For example: diff --git a/docker/.env.example b/docker/.env.example index 46666cf8b..ca656e8a7 100644 --- a/docker/.env.example +++ b/docker/.env.example @@ -49,7 +49,7 @@ PORT=3000 ############################################## STORAGE ##################################################### ############################################################################################################ -# STORAGE_TYPE=local (local | s3 | gcs) +# STORAGE_TYPE=local (local | s3 | gcs | azure) # BLOB_STORAGE_PATH=/your_storage_path/.flowise/storage # S3_STORAGE_BUCKET_NAME=flowise # S3_STORAGE_ACCESS_KEY_ID= @@ -61,6 +61,12 @@ PORT=3000 # GOOGLE_CLOUD_STORAGE_PROJ_ID= # GOOGLE_CLOUD_STORAGE_BUCKET_NAME= # GOOGLE_CLOUD_UNIFORM_BUCKET_ACCESS=true +# Azure Blob Storage (when STORAGE_TYPE=azure) +# Provide EITHER a connection string OR account name + key +# AZURE_BLOB_STORAGE_CONNECTION_STRING= +# AZURE_BLOB_STORAGE_ACCOUNT_NAME= +# AZURE_BLOB_STORAGE_ACCOUNT_KEY= +# AZURE_BLOB_STORAGE_CONTAINER_NAME= ############################################################################################################ diff --git a/docker/docker-compose-queue-prebuilt.yml b/docker/docker-compose-queue-prebuilt.yml index c6379ff5a..8dfff7395 100644 --- a/docker/docker-compose-queue-prebuilt.yml +++ b/docker/docker-compose-queue-prebuilt.yml @@ -67,6 +67,11 @@ services: - GOOGLE_CLOUD_STORAGE_PROJ_ID=${GOOGLE_CLOUD_STORAGE_PROJ_ID} - GOOGLE_CLOUD_STORAGE_BUCKET_NAME=${GOOGLE_CLOUD_STORAGE_BUCKET_NAME} - GOOGLE_CLOUD_UNIFORM_BUCKET_ACCESS=${GOOGLE_CLOUD_UNIFORM_BUCKET_ACCESS} + # Azure Blob Storage (provide EITHER connection string OR account name + key) + - AZURE_BLOB_STORAGE_CONNECTION_STRING=${AZURE_BLOB_STORAGE_CONNECTION_STRING} + - AZURE_BLOB_STORAGE_ACCOUNT_NAME=${AZURE_BLOB_STORAGE_ACCOUNT_NAME} + - AZURE_BLOB_STORAGE_ACCOUNT_KEY=${AZURE_BLOB_STORAGE_ACCOUNT_KEY} + - AZURE_BLOB_STORAGE_CONTAINER_NAME=${AZURE_BLOB_STORAGE_CONTAINER_NAME} # SETTINGS - NUMBER_OF_PROXIES=${NUMBER_OF_PROXIES} diff --git a/docker/docker-compose.yml b/docker/docker-compose.yml index c30f1cc2a..30ced6939 100644 --- a/docker/docker-compose.yml +++ b/docker/docker-compose.yml @@ -52,6 +52,11 @@ services: - GOOGLE_CLOUD_STORAGE_PROJ_ID=${GOOGLE_CLOUD_STORAGE_PROJ_ID} - GOOGLE_CLOUD_STORAGE_BUCKET_NAME=${GOOGLE_CLOUD_STORAGE_BUCKET_NAME} - GOOGLE_CLOUD_UNIFORM_BUCKET_ACCESS=${GOOGLE_CLOUD_UNIFORM_BUCKET_ACCESS} + # Azure Blob Storage (provide EITHER connection string OR account name + key) + - AZURE_BLOB_STORAGE_CONNECTION_STRING=${AZURE_BLOB_STORAGE_CONNECTION_STRING} + - AZURE_BLOB_STORAGE_ACCOUNT_NAME=${AZURE_BLOB_STORAGE_ACCOUNT_NAME} + - AZURE_BLOB_STORAGE_ACCOUNT_KEY=${AZURE_BLOB_STORAGE_ACCOUNT_KEY} + - AZURE_BLOB_STORAGE_CONTAINER_NAME=${AZURE_BLOB_STORAGE_CONTAINER_NAME} # SETTINGS - NUMBER_OF_PROXIES=${NUMBER_OF_PROXIES} diff --git a/docker/worker/.env.example b/docker/worker/.env.example index 36f9d966a..3617d61ea 100644 --- a/docker/worker/.env.example +++ b/docker/worker/.env.example @@ -49,7 +49,7 @@ WORKER_PORT=5566 ############################################## STORAGE ##################################################### ############################################################################################################ -# STORAGE_TYPE=local (local | s3 | gcs) +# STORAGE_TYPE=local (local | s3 | gcs | azure) # BLOB_STORAGE_PATH=/your_storage_path/.flowise/storage # S3_STORAGE_BUCKET_NAME=flowise # S3_STORAGE_ACCESS_KEY_ID= @@ -61,6 +61,12 @@ WORKER_PORT=5566 # GOOGLE_CLOUD_STORAGE_PROJ_ID= # GOOGLE_CLOUD_STORAGE_BUCKET_NAME= # GOOGLE_CLOUD_UNIFORM_BUCKET_ACCESS=true +# Azure Blob Storage (when STORAGE_TYPE=azure) +# Provide EITHER a connection string OR account name + key +# AZURE_BLOB_STORAGE_CONNECTION_STRING= +# AZURE_BLOB_STORAGE_ACCOUNT_NAME= +# AZURE_BLOB_STORAGE_ACCOUNT_KEY= +# AZURE_BLOB_STORAGE_CONTAINER_NAME= ############################################################################################################ diff --git a/docker/worker/docker-compose.yml b/docker/worker/docker-compose.yml index e3370cd8c..4e727022f 100644 --- a/docker/worker/docker-compose.yml +++ b/docker/worker/docker-compose.yml @@ -52,6 +52,11 @@ services: - GOOGLE_CLOUD_STORAGE_PROJ_ID=${GOOGLE_CLOUD_STORAGE_PROJ_ID} - GOOGLE_CLOUD_STORAGE_BUCKET_NAME=${GOOGLE_CLOUD_STORAGE_BUCKET_NAME} - GOOGLE_CLOUD_UNIFORM_BUCKET_ACCESS=${GOOGLE_CLOUD_UNIFORM_BUCKET_ACCESS} + # Azure Blob Storage (provide EITHER connection string OR account name + key) + - AZURE_BLOB_STORAGE_CONNECTION_STRING=${AZURE_BLOB_STORAGE_CONNECTION_STRING} + - AZURE_BLOB_STORAGE_ACCOUNT_NAME=${AZURE_BLOB_STORAGE_ACCOUNT_NAME} + - AZURE_BLOB_STORAGE_ACCOUNT_KEY=${AZURE_BLOB_STORAGE_ACCOUNT_KEY} + - AZURE_BLOB_STORAGE_CONTAINER_NAME=${AZURE_BLOB_STORAGE_CONTAINER_NAME} # SETTINGS - NUMBER_OF_PROXIES=${NUMBER_OF_PROXIES} diff --git a/packages/agentflow/README.md b/packages/agentflow/README.md index fd37a49e2..c58e8d087 100644 --- a/packages/agentflow/README.md +++ b/packages/agentflow/README.md @@ -1,207 +1,207 @@ -# @flowiseai/agentflow - -[![Version](https://img.shields.io/npm/v/@flowiseai/agentflow)](https://www.npmjs.com/package/@flowiseai/agentflow) -[![License](https://img.shields.io/badge/license-Apache--2.0-blue)](https://github.com/FlowiseAI/Flowise/blob/main/LICENSE.md) - -> Embeddable React component for building and visualizing AI agent workflows - -## ⚠️ Status - -**This package is currently under active development.** - -- 🚧 Components are not yet fully functional -- ❌ End-to-end functionality is not complete -- 🔄 Features are still being implemented and tested -- ⚡ APIs may change before stable release -- 📝 Documentation is being updated as development progresses - -**Cannot be used in production. For development and testing purposes only.** - -## Overview - -`@flowiseai/agentflow` is a React-based flow editor for creating AI agent workflows. It provides a visual canvas built on ReactFlow for connecting AI agents, LLMs, tools, and logic nodes. - -## Installation - -```bash -pnpm add @flowiseai/agentflow -``` - -**Peer Dependencies:** - -```bash -pnpm add react react-dom @mui/material @mui/icons-material @emotion/react @emotion/styled reactflow -``` - -## Basic Usage - -```tsx -import { Agentflow } from '@flowiseai/agentflow' - -import '@flowiseai/agentflow/flowise.css' - -export default function App() { - return ( -
- -
- ) -} -``` - -### With Initial Flow Data and Callbacks - -```tsx -import { useRef } from 'react' - -import { Agentflow, type AgentFlowInstance, type FlowData } from '@flowiseai/agentflow' - -import '@flowiseai/agentflow/flowise.css' - -export default function App() { - const ref = useRef(null) - - const initialFlow: FlowData = { - nodes: [ - { - id: 'startAgentflow_0', - type: 'agentflowNode', - position: { x: 100, y: 100 }, - data: { - id: 'startAgentflow_0', - name: 'startAgentflow', - label: 'Start', - color: '#7EE787', - hideInput: true, - outputAnchors: [{ id: 'startAgentflow_0-output-0', name: 'start', label: 'Start', type: 'start' }] - } - } - ], - edges: [], - viewport: { x: 0, y: 0, zoom: 1 } - } - - return ( -
- console.log('Flow changed:', flow)} - onSave={(flow) => console.log('Flow saved:', flow)} - /> -
- ) -} -``` - -## Props - -| Prop | Type | Default | Description | -| -------------------- | ------------------------------------------ | -------------- | --------------------------------------------------------------- | -| `apiBaseUrl` | `string` | **(required)** | Flowise API server endpoint | -| `token` | `string` | — | Authentication token for API calls | -| `initialFlow` | `FlowData` | — | Initial flow data to render (uncontrolled — only used on mount) | -| `components` | `string[]` | — | Restrict which node types appear in the palette | -| `onFlowChange` | `(flow: FlowData) => void` | — | Called when the flow changes (node/edge add, remove, move) | -| `onSave` | `(flow: FlowData) => void` | — | Called when the user triggers a save | -| `onFlowGenerated` | `(flow: FlowData) => void` | — | Called when a flow is generated via AI | -| `isDarkMode` | `boolean` | `false` | Use dark mode theme | -| `readOnly` | `boolean` | `false` | Disable editing (nodes not draggable/connectable) | -| `showDefaultHeader` | `boolean` | `true` | Show built-in header (ignored if `renderHeader` provided) | -| `showDefaultPalette` | `boolean` | `true` | Show built-in node palette | -| `enableGenerator` | `boolean` | `true` | Show the AI flow generator button | -| `renderHeader` | `(props: HeaderRenderProps) => ReactNode` | — | Custom header renderer | -| `renderNodePalette` | `(props: PaletteRenderProps) => ReactNode` | — | Custom node palette renderer | - -### Imperative Methods (via `ref`) - -| Method | Return Type | Description | -| ------------------------ | ------------------------- | ------------------------------------- | -| `getFlow()` | `FlowData` | Get current flow data | -| `toJSON()` | `string` | Export flow as JSON string | -| `validate()` | `ValidationResult` | Validate the current flow | -| `fitView()` | `void` | Fit all nodes into view | -| `clear()` | `void` | Remove all nodes and edges | -| `addNode(nodeData)` | `void` | Add a node (`Partial`) | -| `getReactFlowInstance()` | `ReactFlowInstance\|null` | Get the underlying ReactFlow instance | - -### Design Note - -`` is an **uncontrolled component**. The `initialFlow` prop seeds the canvas state on mount, but the component owns its own state afterward. Use the `ref` for imperative access and `onFlowChange` to observe changes. - -## Development - -```bash -# Install dependencies -pnpm install - -# Build the package -pnpm build - -# Run tests -pnpm test - -# Run examples -cd examples && pnpm install && pnpm dev -``` - -Visit the [examples](./examples) directory for more usage patterns. See [TESTS.md](./TESTS.md) for the full test plan and coverage status. - -## Publishing - -### Version Update - -Bump the version in `package.json` before publishing. Use `npm version` to update the version and create a git tag: - -```bash -# Prerelease (for testing) -npm version prerelease --preid=dev # 0.0.0-dev.1 → 0.0.0-dev.2 - -# Patch / Minor / Major (for stable releases) -npm version patch # 0.0.1 -npm version minor # 0.1.0 -npm version major # 1.0.0 -``` - -### Verify Before Publishing - -```bash -# Build and check the tarball contents -pnpm build -npm pack --dry-run - -# Full publish dry-run (runs prepublishOnly + simulates upload) -npm publish --dry-run -``` - -### Publish - -```bash -# Prerelease — tagged so `npm install @flowiseai/agentflow` won't pick it up -npm publish --tag dev - -# Stable release — gets the `latest` tag -npm publish -``` - -> The `prepublishOnly` script automatically runs `clean` and `build` before every publish, so stale dist files are never uploaded. - -## Documentation - -- [ARCHITECTURE.md](./ARCHITECTURE.md) - Internal architecture and design patterns -- [TESTS.md](./TESTS.md) - Test plan, coverage tiers, and configuration -- [Examples](./examples/README.md) - Usage examples and demos - -## Contributing - -This package follows a feature-based architecture with clear separation of concerns. See [ARCHITECTURE.md](./ARCHITECTURE.md) for details on the project structure and development guidelines. - -## License - -Apache-2.0 — see the repository root [LICENSE.md](https://github.com/FlowiseAI/Flowise/blob/main/LICENSE.md) for details. - ---- - -Part of the [Flowise](https://github.com/FlowiseAI/Flowise) ecosystem +# @flowiseai/agentflow + +[![Version](https://img.shields.io/npm/v/@flowiseai/agentflow)](https://www.npmjs.com/package/@flowiseai/agentflow) +[![License](https://img.shields.io/badge/license-Apache--2.0-blue)](https://github.com/FlowiseAI/Flowise/blob/main/LICENSE.md) + +> Embeddable React component for building and visualizing AI agent workflows + +## ⚠️ Status + +**This package is currently under active development.** + +- 🚧 Components are not yet fully functional +- ❌ End-to-end functionality is not complete +- 🔄 Features are still being implemented and tested +- ⚡ APIs may change before stable release +- 📝 Documentation is being updated as development progresses + +**Cannot be used in production. For development and testing purposes only.** + +## Overview + +`@flowiseai/agentflow` is a React-based flow editor for creating AI agent workflows. It provides a visual canvas built on ReactFlow for connecting AI agents, LLMs, tools, and logic nodes. + +## Installation + +```bash +pnpm add @flowiseai/agentflow +``` + +**Peer Dependencies:** + +```bash +pnpm add react react-dom @mui/material @mui/icons-material @emotion/react @emotion/styled reactflow +``` + +## Basic Usage + +```tsx +import { Agentflow } from '@flowiseai/agentflow' + +import '@flowiseai/agentflow/flowise.css' + +export default function App() { + return ( +
+ +
+ ) +} +``` + +### With Initial Flow Data and Callbacks + +```tsx +import { useRef } from 'react' + +import { Agentflow, type AgentFlowInstance, type FlowData } from '@flowiseai/agentflow' + +import '@flowiseai/agentflow/flowise.css' + +export default function App() { + const ref = useRef(null) + + const initialFlow: FlowData = { + nodes: [ + { + id: 'startAgentflow_0', + type: 'agentflowNode', + position: { x: 100, y: 100 }, + data: { + id: 'startAgentflow_0', + name: 'startAgentflow', + label: 'Start', + color: '#7EE787', + hideInput: true, + outputAnchors: [{ id: 'startAgentflow_0-output-0', name: 'start', label: 'Start', type: 'start' }] + } + } + ], + edges: [], + viewport: { x: 0, y: 0, zoom: 1 } + } + + return ( +
+ console.log('Flow changed:', flow)} + onSave={(flow) => console.log('Flow saved:', flow)} + /> +
+ ) +} +``` + +## Props + +| Prop | Type | Default | Description | +| -------------------- | ------------------------------------------ | -------------- | --------------------------------------------------------------- | +| `apiBaseUrl` | `string` | **(required)** | Flowise API server endpoint | +| `token` | `string` | — | Authentication token for API calls | +| `initialFlow` | `FlowData` | — | Initial flow data to render (uncontrolled — only used on mount) | +| `components` | `string[]` | — | Restrict which node types appear in the palette | +| `onFlowChange` | `(flow: FlowData) => void` | — | Called when the flow changes (node/edge add, remove, move) | +| `onSave` | `(flow: FlowData) => void` | — | Called when the user triggers a save | +| `onFlowGenerated` | `(flow: FlowData) => void` | — | Called when a flow is generated via AI | +| `isDarkMode` | `boolean` | `false` | Use dark mode theme | +| `readOnly` | `boolean` | `false` | Disable editing (nodes not draggable/connectable) | +| `showDefaultHeader` | `boolean` | `true` | Show built-in header (ignored if `renderHeader` provided) | +| `showDefaultPalette` | `boolean` | `true` | Show built-in node palette | +| `enableGenerator` | `boolean` | `true` | Show the AI flow generator button | +| `renderHeader` | `(props: HeaderRenderProps) => ReactNode` | — | Custom header renderer | +| `renderNodePalette` | `(props: PaletteRenderProps) => ReactNode` | — | Custom node palette renderer | + +### Imperative Methods (via `ref`) + +| Method | Return Type | Description | +| ------------------------ | ------------------------- | ------------------------------------- | +| `getFlow()` | `FlowData` | Get current flow data | +| `toJSON()` | `string` | Export flow as JSON string | +| `validate()` | `ValidationResult` | Validate the current flow | +| `fitView()` | `void` | Fit all nodes into view | +| `clear()` | `void` | Remove all nodes and edges | +| `addNode(nodeData)` | `void` | Add a node (`Partial`) | +| `getReactFlowInstance()` | `ReactFlowInstance\|null` | Get the underlying ReactFlow instance | + +### Design Note + +`` is an **uncontrolled component**. The `initialFlow` prop seeds the canvas state on mount, but the component owns its own state afterward. Use the `ref` for imperative access and `onFlowChange` to observe changes. + +## Development + +```bash +# Install dependencies +pnpm install + +# Build the package +pnpm build + +# Run tests +pnpm test + +# Run examples +cd examples && pnpm install && pnpm dev +``` + +Visit the [examples](./examples) directory for more usage patterns. See [TESTS.md](./TESTS.md) for the full test plan and coverage status. + +## Publishing + +### Version Update + +Bump the version in `package.json` before publishing. Use `npm version` to update the version and create a git tag: + +```bash +# Prerelease (for testing) +npm version prerelease --preid=dev # 0.0.0-dev.1 → 0.0.0-dev.2 + +# Patch / Minor / Major (for stable releases) +npm version patch # 0.0.1 +npm version minor # 0.1.0 +npm version major # 1.0.0 +``` + +### Verify Before Publishing + +```bash +# Build and check the tarball contents +pnpm build +npm pack --dry-run + +# Full publish dry-run (runs prepublishOnly + simulates upload) +npm publish --dry-run +``` + +### Publish + +```bash +# Prerelease — tagged so `npm install @flowiseai/agentflow` won't pick it up +npm publish --tag dev + +# Stable release — gets the `latest` tag +npm publish +``` + +> The `prepublishOnly` script automatically runs `clean` and `build` before every publish, so stale dist files are never uploaded. + +## Documentation + +- [ARCHITECTURE.md](./ARCHITECTURE.md) - Internal architecture and design patterns +- [TESTS.md](./TESTS.md) - Test plan, coverage tiers, and configuration +- [Examples](./examples/README.md) - Usage examples and demos + +## Contributing + +This package follows a feature-based architecture with clear separation of concerns. See [ARCHITECTURE.md](./ARCHITECTURE.md) for details on the project structure and development guidelines. + +## License + +Apache-2.0 — see the repository root [LICENSE.md](https://github.com/FlowiseAI/Flowise/blob/main/LICENSE.md) for details. + +--- + +Part of the [Flowise](https://github.com/FlowiseAI/Flowise) ecosystem diff --git a/packages/components/package.json b/packages/components/package.json index 1071e9da0..c8cce49b0 100644 --- a/packages/components/package.json +++ b/packages/components/package.json @@ -42,6 +42,7 @@ "@gomomento/sdk-core": "^1.51.1", "@google-ai/generativelanguage": "^2.5.0", "@google-cloud/storage": "^7.15.2", + "@azure/storage-blob": "^12.29.0", "@google/generative-ai": "^0.24.0", "@grpc/grpc-js": "^1.10.10", "@huggingface/inference": "^4.13.2", diff --git a/packages/components/src/index.ts b/packages/components/src/index.ts index c2c737fc0..6b77c943e 100644 --- a/packages/components/src/index.ts +++ b/packages/components/src/index.ts @@ -9,6 +9,7 @@ export * from './utils' export * from './speechToText' export * from './textToSpeech' export * from './storageUtils' +export * from './storage' export * from './handler' export * from '../evaluation/EvaluationRunner' export * from './followUpPrompts' diff --git a/packages/components/src/storage/AzureBlobStorageProvider.ts b/packages/components/src/storage/AzureBlobStorageProvider.ts new file mode 100644 index 000000000..43d720fe9 --- /dev/null +++ b/packages/components/src/storage/AzureBlobStorageProvider.ts @@ -0,0 +1,331 @@ +import { BlobServiceClient, ContainerClient, StorageSharedKeyCredential } from '@azure/storage-blob' +import multer from 'multer' +import { MulterAzureStorage } from 'multer-azure-blob-storage' +import { v4 as uuidv4 } from 'uuid' +import { BaseStorageProvider } from './BaseStorageProvider' +import { FileInfo, StorageResult, StorageSizeResult } from './IStorageProvider' +import { winstonAzureBlob } from 'winston-azure-blob' + +/** + * Extends MulterAzureStorage to set file.path from file.blobName after upload. + * The server expects file.path (local/GCS) or file.key (S3) but multer-azure-blob-storage + * only sets file.blobName. This subclass bridges that gap. + */ +class MulterAzureStorageWithPath extends MulterAzureStorage { + _handleFile(req: any, file: any, cb: (error?: any, info?: any) => void): Promise { + return super._handleFile(req, file, (err: any, info: any) => { + if (!err && info) { + info.path = info.blobName + } + cb(err, info) + }) + } +} + +export class AzureBlobStorageProvider extends BaseStorageProvider { + private containerClient: ContainerClient + private containerName: string + + constructor() { + super() + const config = this.initAzureConfig() + this.containerClient = config.containerClient + this.containerName = config.containerName + } + + private initAzureConfig(): { + containerClient: ContainerClient + containerName: string + } { + const connectionString = process.env.AZURE_BLOB_STORAGE_CONNECTION_STRING + const accountName = process.env.AZURE_BLOB_STORAGE_ACCOUNT_NAME + const accountKey = process.env.AZURE_BLOB_STORAGE_ACCOUNT_KEY + const containerName = process.env.AZURE_BLOB_STORAGE_CONTAINER_NAME + + if (!containerName || containerName.trim() === '') { + throw new Error('AZURE_BLOB_STORAGE_CONTAINER_NAME env variable is required') + } + + let blobServiceClient: BlobServiceClient + + // Authenticate either using connection string or account name and key + if (connectionString && connectionString.trim() !== '') { + blobServiceClient = BlobServiceClient.fromConnectionString(connectionString) + } else if (accountName && accountName.trim() !== '' && accountKey && accountKey.trim() !== '') { + const sharedKeyCredential = new StorageSharedKeyCredential(accountName, accountKey) + blobServiceClient = new BlobServiceClient(`https://${accountName}.blob.core.windows.net`, sharedKeyCredential) + } else { + throw new Error( + 'Azure Blob Storage configuration is missing. Provide AZURE_BLOB_STORAGE_CONNECTION_STRING or AZURE_BLOB_STORAGE_ACCOUNT_NAME + AZURE_BLOB_STORAGE_ACCOUNT_KEY' + ) + } + + const containerClient = blobServiceClient.getContainerClient(containerName) + return { containerClient, containerName } + } + + getStorageType(): string { + return 'azure' + } + + getConfig(): any { + return { + containerName: this.containerName + } + } + + async addBase64FilesToStorage(fileBase64: string, chatflowid: string, fileNames: string[], orgId: string): Promise { + // Validate chatflowid + this.validateChatflowId(chatflowid) + this.validatePathSecurity(chatflowid) + + const splitDataURI = fileBase64.split(',') + const filename = splitDataURI.pop()?.split(':')[1] ?? '' + const bf = Buffer.from(splitDataURI.pop() || '', 'base64') + const mime = splitDataURI[0].split(':')[1].split(';')[0] + + const sanitizedFilename = this.sanitizeFilename(filename) + const blobName = `${orgId}/${chatflowid}/${sanitizedFilename}` + + const blockBlobClient = this.containerClient.getBlockBlobClient(blobName) + await blockBlobClient.upload(bf, bf.length, { + blobHTTPHeaders: { blobContentType: mime } + }) + + fileNames.push(sanitizedFilename) + const totalSize = await this.getStorageSize(orgId) + + return { path: 'FILE-STORAGE::' + JSON.stringify(fileNames), totalSize: totalSize / 1024 / 1024 } + } + + async addArrayFilesToStorage( + mime: string, + bf: Buffer, + fileName: string, + fileNames: string[], + ...paths: string[] + ): Promise { + const sanitizedFilename = this.sanitizeFilename(fileName) + + let blobName = paths.reduce((acc, cur) => acc + '/' + cur, '') + '/' + sanitizedFilename + if (blobName.startsWith('/')) { + blobName = blobName.substring(1) + } + + const blockBlobClient = this.containerClient.getBlockBlobClient(blobName) + await blockBlobClient.upload(bf, bf.length, { + blobHTTPHeaders: { blobContentType: mime } + }) + fileNames.push(sanitizedFilename) + + const totalSize = await this.getStorageSize(paths[0]) + + return { path: 'FILE-STORAGE::' + JSON.stringify(fileNames), totalSize: totalSize / 1024 / 1024 } + } + + async addSingleFileToStorage(mime: string, bf: Buffer, fileName: string, ...paths: string[]): Promise { + const sanitizedFilename = this.sanitizeFilename(fileName) + + let blobName = paths.reduce((acc, cur) => acc + '/' + cur, '') + '/' + sanitizedFilename + if (blobName.startsWith('/')) { + blobName = blobName.substring(1) + } + + const blockBlobClient = this.containerClient.getBlockBlobClient(blobName) + await blockBlobClient.upload(bf, bf.length, { + blobHTTPHeaders: { blobContentType: mime } + }) + + const totalSize = await this.getStorageSize(paths[0]) + + return { path: 'FILE-STORAGE::' + sanitizedFilename, totalSize: totalSize / 1024 / 1024 } + } + + async getFileFromUpload(filePath: string): Promise { + let blobName = filePath + // remove the first '/' if it exists + if (blobName.startsWith('/')) { + blobName = blobName.substring(1) + } + + const blockBlobClient = this.containerClient.getBlockBlobClient(blobName) + const downloadResponse = await blockBlobClient.downloadToBuffer() + return downloadResponse + } + + async getFileFromStorage(file: string, ...paths: string[]): Promise { + const sanitizedFilename = this.sanitizeFilename(file) + + let blobName = paths.reduce((acc, cur) => acc + '/' + cur, '') + '/' + sanitizedFilename + if (blobName.startsWith('/')) { + blobName = blobName.substring(1) + } + + const blockBlobClient = this.containerClient.getBlockBlobClient(blobName) + const buffer = await blockBlobClient.downloadToBuffer() + return buffer + } + + async streamStorageFile(chatflowId: string, chatId: string, fileName: string, orgId: string): Promise { + // Validate chatflowId and chatId + this.validateChatflowId(chatflowId) + this.validatePathSecurity(chatflowId, chatId) + + const sanitizedFilename = this.sanitizeFilename(fileName) + const blobName = `${orgId}/${chatflowId}/${chatId}/${sanitizedFilename}` + + const blockBlobClient = this.containerClient.getBlockBlobClient(blobName) + const buffer = await blockBlobClient.downloadToBuffer() + return buffer + } + + async getFilesListFromStorage(...paths: string[]): Promise { + let prefix = paths.reduce((acc, cur) => acc + '/' + cur, '') + if (prefix.startsWith('/')) { + prefix = prefix.substring(1) + } + + const filesList: FileInfo[] = [] + for await (const blob of this.containerClient.listBlobsFlat({ prefix })) { + filesList.push({ + name: blob.name.split('/').pop() || '', + path: blob.name, + size: blob.properties.contentLength || 0 + }) + } + return filesList + } + + async removeFilesFromStorage(...paths: string[]): Promise { + let prefix = paths.reduce((acc, cur) => acc + '/' + cur, '') + if (prefix.startsWith('/')) { + prefix = prefix.substring(1) + } + + // Delete all blobs with the prefix + for await (const blob of this.containerClient.listBlobsFlat({ prefix })) { + await this.containerClient.deleteBlob(blob.name) + } + + const totalSize = await this.getStorageSize(paths[0]) + return { totalSize: totalSize / 1024 / 1024 } + } + + async removeSpecificFileFromUpload(filePath: string): Promise { + let blobName = filePath + if (blobName.startsWith('/')) { + blobName = blobName.substring(1) + } + + await this.containerClient.deleteBlob(blobName) + } + + async removeSpecificFileFromStorage(...paths: string[]): Promise { + let blobName = paths.reduce((acc, cur) => acc + '/' + cur, '') + if (blobName.startsWith('/')) { + blobName = blobName.substring(1) + } + + const blockBlobClient = this.containerClient.getBlockBlobClient(blobName) + if (await blockBlobClient.exists()) { + await blockBlobClient.delete() + } + + const totalSize = await this.getStorageSize(paths[0]) + return { totalSize: totalSize / 1024 / 1024 } + } + + async removeFolderFromStorage(...paths: string[]): Promise { + let prefix = paths.reduce((acc, cur) => acc + '/' + cur, '') + if (prefix.startsWith('/')) { + prefix = prefix.substring(1) + } + + // Delete all blobs with the prefix + for await (const blob of this.containerClient.listBlobsFlat({ prefix })) { + await this.containerClient.deleteBlob(blob.name) + } + + const totalSize = await this.getStorageSize(paths[0]) + return { totalSize: totalSize / 1024 / 1024 } + } + + async getStorageSize(orgId: string): Promise { + if (!orgId) return 0 + + let totalSize = 0 + for await (const blob of this.containerClient.listBlobsFlat({ prefix: orgId })) { + totalSize += blob.properties.contentLength || 0 + } + + return totalSize + } + + getMulterStorage(): multer.Multer { + const connectionString = process.env.AZURE_BLOB_STORAGE_CONNECTION_STRING + + let storageConfig: any = { + containerName: this.containerName, + blobName: async (_req: any, file: any) => `uploads/${uuidv4()}/${file.originalname}` + } + + // Use connection string if available, otherwise use account name/key + if (connectionString && connectionString.trim() !== '') { + storageConfig.connectionString = connectionString + } else { + storageConfig.accountName = process.env.AZURE_BLOB_STORAGE_ACCOUNT_NAME + storageConfig.accessKey = process.env.AZURE_BLOB_STORAGE_ACCOUNT_KEY + } + + const azureStorage = new MulterAzureStorageWithPath(storageConfig) + return multer({ storage: azureStorage }) + } + + getLoggerTransports(logType: 'server' | 'error' | 'requests'): any[] { + const connectionString = process.env.AZURE_BLOB_STORAGE_CONNECTION_STRING + const accountName = process.env.AZURE_BLOB_STORAGE_ACCOUNT_NAME + const accountKey = process.env.AZURE_BLOB_STORAGE_ACCOUNT_KEY + + let baseConfig: any = { containerName: this.containerName } + + // Support both connection string and account name/key authentication + if (connectionString && connectionString.trim() !== '') { + baseConfig.account = { connectionString } + } else { + baseConfig.account = { name: accountName, key: accountKey } + } + + if (logType === 'server') { + return [ + winstonAzureBlob({ + ...baseConfig, + blobName: 'logs/server/server', + rotatePeriod: 'YYYY-MM-DD', + extension: '.log', + level: 'info' + }) + ] + } else if (logType === 'error') { + return [ + winstonAzureBlob({ + ...baseConfig, + blobName: 'logs/error/server-error', + rotatePeriod: 'YYYY-MM-DD', + extension: '.log', + level: 'error' + }) + ] + } else if (logType === 'requests') { + return [ + winstonAzureBlob({ + ...baseConfig, + blobName: 'logs/requests/server-requests', + rotatePeriod: 'YYYY-MM-DD', + extension: '.log.jsonl', + level: 'debug' + }) + ] + } + return [] + } +} diff --git a/packages/components/src/storage/BaseStorageProvider.ts b/packages/components/src/storage/BaseStorageProvider.ts new file mode 100644 index 000000000..57bbca693 --- /dev/null +++ b/packages/components/src/storage/BaseStorageProvider.ts @@ -0,0 +1,100 @@ +import path from 'node:path' +import { IStorageProvider, FileInfo, StorageResult, StorageSizeResult } from './IStorageProvider' +import sanitize from 'sanitize-filename' +import { getUserHome } from '../utils' +import { isPathTraversal, isUnsafeFilePath, isValidUUID } from '../validator' +import fs from 'node:fs' + +export abstract class BaseStorageProvider implements IStorageProvider { + protected storagePath: string + + constructor() { + this.storagePath = this.getStoragePath() + } + + abstract getStorageType(): string + abstract getConfig(): any + abstract addBase64FilesToStorage(fileBase64: string, chatflowid: string, fileNames: string[], orgId: string): Promise + abstract addArrayFilesToStorage( + mime: string, + bf: Buffer, + fileName: string, + fileNames: string[], + ...paths: string[] + ): Promise + abstract addSingleFileToStorage(mime: string, bf: Buffer, fileName: string, ...paths: string[]): Promise + abstract getFileFromUpload(filePath: string): Promise + abstract getFileFromStorage(file: string, ...paths: string[]): Promise + abstract getFilesListFromStorage(...paths: string[]): Promise + abstract streamStorageFile( + chatflowId: string, + chatId: string, + fileName: string, + orgId: string + ): Promise + abstract removeFilesFromStorage(...paths: string[]): Promise + abstract removeSpecificFileFromUpload(filePath: string): Promise + abstract removeSpecificFileFromStorage(...paths: string[]): Promise + abstract removeFolderFromStorage(...paths: string[]): Promise + abstract getStorageSize(orgId: string): Promise + abstract getMulterStorage(): any + abstract getLoggerTransports(logType: 'server' | 'error' | 'requests', config?: any): any[] + + /** + * Shared utility for sanitizing filenames to prevent path traversal and other issues + */ + protected sanitizeFilename(filename: string): string { + if (!filename || isUnsafeFilePath(filename)) { + throw new Error('Invalid or unsafe fileName detected') + } + const sanitizedFilename = sanitize(filename) + // Remove leading dots to prevent hidden files or relative path jumps + const cleaned = sanitizedFilename.replace(/^\.+/, '') + if (!cleaned || cleaned.includes('/') || cleaned.includes('\\')) { + throw new Error('Invalid filename after sanitization') + } + return cleaned + } + + /** + * Shared utility for getting the base storage path + */ + protected getStoragePath(): string { + const storagePath = process.env.BLOB_STORAGE_PATH + ? path.join(process.env.BLOB_STORAGE_PATH) + : path.join(getUserHome(), '.flowise', 'storage') + + if (!fs.existsSync(storagePath)) { + fs.mkdirSync(storagePath, { recursive: true }) + } + return storagePath + } + + /** + * Shared utility for validating chatflowId format (UUID) + */ + protected validateChatflowId(chatflowId: string): void { + if (!chatflowId || !isValidUUID(chatflowId)) { + throw new Error('Invalid chatflowId format - must be a valid UUID') + } + } + + /** + * Shared utility for checking path traversal attempts + */ + protected validatePathSecurity(...paths: string[]): void { + for (const p of paths) { + if (p && isPathTraversal(p)) { + throw new Error('Invalid path characters detected') + } + } + } + + /** + * Shared utility for building a storage path from components + */ + protected buildPath(...paths: string[]): string { + const sanitizedPaths = paths.filter((p) => p && typeof p === 'string').map((p) => this.sanitizeFilename(p)) + return path.join(this.storagePath, ...sanitizedPaths) + } +} diff --git a/packages/components/src/storage/GCSStorageProvider.ts b/packages/components/src/storage/GCSStorageProvider.ts new file mode 100644 index 000000000..0aa06c949 --- /dev/null +++ b/packages/components/src/storage/GCSStorageProvider.ts @@ -0,0 +1,374 @@ +import { Storage, Bucket } from '@google-cloud/storage' +import { LoggingWinston } from '@google-cloud/logging-winston' +import multer from 'multer' +import { v4 as uuidv4 } from 'uuid' +import { BaseStorageProvider } from './BaseStorageProvider' +import { FileInfo, StorageResult, StorageSizeResult } from './IStorageProvider' + +import MulterGoogleCloudStorage from 'multer-cloud-storage' + +export class GCSStorageProvider extends BaseStorageProvider { + private bucket: Bucket + private bucketName: string + private projectId: string | undefined + private keyFilename: string | undefined + + constructor() { + super() + const config = this.initGCSConfig() + + this.bucket = config.bucket + this.bucketName = config.bucketName + this.projectId = config.projectId + this.keyFilename = config.keyFilename + } + + private initGCSConfig(): { + bucket: Bucket + bucketName: string + projectId: string | undefined + keyFilename: string | undefined + } { + const keyFilename = process.env.GOOGLE_CLOUD_STORAGE_CREDENTIAL + const projectId = process.env.GOOGLE_CLOUD_STORAGE_PROJ_ID + const bucketName = process.env.GOOGLE_CLOUD_STORAGE_BUCKET_NAME + + if (!bucketName) { + throw new Error('GOOGLE_CLOUD_STORAGE_BUCKET_NAME env variable is required') + } + + const storageConfig = { + ...(keyFilename ? { keyFilename } : {}), + ...(projectId ? { projectId } : {}) + } + + const storage = new Storage(storageConfig) + const bucket = storage.bucket(bucketName) + return { bucket, bucketName, projectId, keyFilename } + } + + getStorageType(): string { + return 'gcs' + } + + getConfig(): any { + return { + bucketName: this.bucketName, + projectId: this.projectId + } + } + + private normalizePath(p: string): string { + return p.replace(/\\/g, '/') + } + + async addBase64FilesToStorage(fileBase64: string, chatflowid: string, fileNames: string[], orgId: string): Promise { + // Validate chatflowid + this.validateChatflowId(chatflowid) + this.validatePathSecurity(chatflowid) + + const splitDataURI = fileBase64.split(',') + const filename = splitDataURI.pop()?.split(':')[1] ?? '' + const bf = Buffer.from(splitDataURI.pop() || '', 'base64') + const mime = splitDataURI[0].split(':')[1].split(';')[0] + + const sanitizedFilename = this.sanitizeFilename(filename) + const normalizedChatflowid = this.normalizePath(chatflowid) + const normalizedFilename = this.normalizePath(sanitizedFilename) + const filePath = `${orgId}/${normalizedChatflowid}/${normalizedFilename}` + + const file = this.bucket.file(filePath) + await new Promise((resolve, reject) => { + file.createWriteStream({ contentType: mime, metadata: { contentEncoding: 'base64' } }) + .on('error', (err) => reject(err)) + .on('finish', () => resolve()) + .end(bf) + }) + + fileNames.push(sanitizedFilename) + const totalSize = await this.getStorageSize(orgId) + + return { path: 'FILE-STORAGE::' + JSON.stringify(fileNames), totalSize: totalSize / 1024 / 1024 } + } + + async addArrayFilesToStorage( + mime: string, + bf: Buffer, + fileName: string, + fileNames: string[], + ...paths: string[] + ): Promise { + const sanitizedFilename = this.sanitizeFilename(fileName) + const normalizedPaths = paths.map((p) => this.normalizePath(p)) + const normalizedFilename = this.normalizePath(sanitizedFilename) + const filePath = [...normalizedPaths, normalizedFilename].join('/') + + const file = this.bucket.file(filePath) + await new Promise((resolve, reject) => { + file.createWriteStream() + .on('error', (err) => reject(err)) + .on('finish', () => resolve()) + .end(bf) + }) + + fileNames.push(sanitizedFilename) + const totalSize = await this.getStorageSize(paths[0]) + + return { path: 'FILE-STORAGE::' + JSON.stringify(fileNames), totalSize: totalSize / 1024 / 1024 } + } + + async addSingleFileToStorage(mime: string, bf: Buffer, fileName: string, ...paths: string[]): Promise { + const sanitizedFilename = this.sanitizeFilename(fileName) + const normalizedPaths = paths.map((p) => this.normalizePath(p)) + const normalizedFilename = this.normalizePath(sanitizedFilename) + const filePath = [...normalizedPaths, normalizedFilename].join('/') + + const file = this.bucket.file(filePath) + await new Promise((resolve, reject) => { + file.createWriteStream({ contentType: mime, metadata: { contentEncoding: 'base64' } }) + .on('error', (err) => reject(err)) + .on('finish', () => resolve()) + .end(bf) + }) + + const totalSize = await this.getStorageSize(paths[0]) + return { path: 'FILE-STORAGE::' + sanitizedFilename, totalSize: totalSize / 1024 / 1024 } + } + + async getFileFromUpload(filePath: string): Promise { + const file = this.bucket.file(filePath) + const [buffer] = await file.download() + return buffer + } + + async getFileFromStorage(file: string, ...paths: string[]): Promise { + const sanitizedFilename = this.sanitizeFilename(file) + const normalizedPaths = paths.map((p) => this.normalizePath(p)) + const normalizedFilename = this.normalizePath(sanitizedFilename) + const filePath = [...normalizedPaths, normalizedFilename].join('/') + + try { + const gcsFile = this.bucket.file(filePath) + const [buffer] = await gcsFile.download() + return buffer + } catch (error) { + // Fallback: Check if file exists without the first path element (likely orgId) + if (normalizedPaths.length > 1) { + const fallbackPaths = normalizedPaths.slice(1) + const fallbackPath = [...fallbackPaths, normalizedFilename].join('/') + + try { + const fallbackFile = this.bucket.file(fallbackPath) + const [buffer] = await fallbackFile.download() + + // Move to correct location with orgId + const gcsFile = this.bucket.file(filePath) + await new Promise((resolve, reject) => { + gcsFile + .createWriteStream() + .on('error', (err) => reject(err)) + .on('finish', () => resolve()) + .end(buffer) + }) + + // Delete the old file + await fallbackFile.delete() + + // Check if the directory is empty and delete recursively if needed + if (fallbackPaths.length > 0) { + await this.cleanEmptyGCSFolders(fallbackPaths[0]) + } + + return buffer + } catch (fallbackError) { + throw error + } + } else { + throw error + } + } + } + + async streamStorageFile(chatflowId: string, chatId: string, fileName: string, orgId: string): Promise { + // Validate chatflowId and chatId + this.validateChatflowId(chatflowId) + this.validatePathSecurity(chatflowId, chatId) + + const sanitizedFilename = this.sanitizeFilename(fileName) + const normalizedChatflowId = this.normalizePath(chatflowId) + const normalizedChatId = this.normalizePath(chatId) + const normalizedFilename = this.normalizePath(sanitizedFilename) + const filePath = `${orgId}/${normalizedChatflowId}/${normalizedChatId}/${normalizedFilename}` + + try { + const [buffer] = await this.bucket.file(filePath).download() + return buffer + } catch (error) { + // Fallback: Check if file exists without orgId + const fallbackPath = `${normalizedChatflowId}/${normalizedChatId}/${normalizedFilename}` + try { + const fallbackFile = this.bucket.file(fallbackPath) + const [buffer] = await fallbackFile.download() + + // If found, copy to correct location with orgId + if (buffer) { + const file = this.bucket.file(filePath) + await new Promise((resolve, reject) => { + file.createWriteStream() + .on('error', (err) => reject(err)) + .on('finish', () => resolve()) + .end(buffer) + }) + + // Delete the old file + await fallbackFile.delete() + + // Check if the directory is empty and delete recursively if needed + await this.cleanEmptyGCSFolders(normalizedChatflowId) + + return buffer + } + } catch (fallbackError) { + throw new Error(`File ${fileName} not found`) + } + } + } + + async getFilesListFromStorage(...paths: string[]): Promise { + const normalizedPaths = paths.map((p) => this.normalizePath(p)) + const prefix = normalizedPaths.join('/') + + const [files] = await this.bucket.getFiles({ prefix }) + + return files.map((file) => ({ + name: file.name.split('/').pop() || '', + path: file.name, + size: typeof file.metadata.size === 'string' ? parseInt(file.metadata.size, 10) || 0 : file.metadata.size || 0 + })) + } + + async removeFilesFromStorage(...paths: string[]): Promise { + const normalizedPath = paths.map((p) => this.normalizePath(p)).join('/') + await this.bucket.deleteFiles({ prefix: `${normalizedPath}/` }) + + const totalSize = await this.getStorageSize(paths[0]) + return { totalSize: totalSize / 1024 / 1024 } + } + + async removeSpecificFileFromUpload(filePath: string): Promise { + await this.bucket.file(filePath).delete() + } + + async removeSpecificFileFromStorage(...paths: string[]): Promise { + const fileName = paths.pop() + if (fileName) { + const sanitizedFilename = this.sanitizeFilename(fileName) + paths.push(sanitizedFilename) + } + const normalizedPath = paths.map((p) => this.normalizePath(p)).join('/') + await this.bucket.file(normalizedPath).delete() + + const totalSize = await this.getStorageSize(paths[0]) + return { totalSize: totalSize / 1024 / 1024 } + } + + async removeFolderFromStorage(...paths: string[]): Promise { + const normalizedPath = paths.map((p) => this.normalizePath(p)).join('/') + await this.bucket.deleteFiles({ prefix: `${normalizedPath}/` }) + + const totalSize = await this.getStorageSize(paths[0]) + return { totalSize: totalSize / 1024 / 1024 } + } + + private async cleanEmptyGCSFolders(prefix: string): Promise { + try { + if (!prefix) return + + const [files] = await this.bucket.getFiles({ + prefix: prefix + '/', + delimiter: '/' + }) + + if (files.length === 0) { + try { + await this.bucket.file(prefix + '/').delete() + } catch (err) { + // Folder marker might not exist, ignore + } + + const parentPrefix = prefix.substring(0, prefix.lastIndexOf('/')) + if (parentPrefix) { + await this.cleanEmptyGCSFolders(parentPrefix) + } + } + } catch (error) { + console.error('Error cleaning empty GCS folders:', error) + } + } + + async getStorageSize(orgId: string): Promise { + if (!orgId) return 0 + + const [files] = await this.bucket.getFiles({ prefix: orgId }) + let totalSize = 0 + + for (const file of files) { + const size = file.metadata.size + if (typeof size === 'string') { + totalSize += parseInt(size, 10) || 0 + } else if (typeof size === 'number') { + totalSize += size + } + } + + return totalSize + } + + getMulterStorage(): multer.Multer { + return multer({ + storage: new MulterGoogleCloudStorage({ + projectId: this.projectId, + bucket: this.bucketName, + keyFilename: this.keyFilename, + uniformBucketLevelAccess: Boolean(process.env.GOOGLE_CLOUD_UNIFORM_BUCKET_ACCESS) ?? true, + destination: `uploads/${uuidv4()}` + }) + }) + } + + getLoggerTransports(logType: 'server' | 'error' | 'requests'): any[] { + const gcsConfig = { + projectId: this.projectId, + keyFilename: this.keyFilename, + defaultCallback: (err: any) => { + if (err) { + console.error('Error logging to GCS: ' + err) + } + } + } + + if (logType === 'server') { + return [ + new LoggingWinston({ + ...gcsConfig, + logName: 'server' + }) + ] + } else if (logType === 'error') { + return [ + new LoggingWinston({ + ...gcsConfig, + logName: 'error' + }) + ] + } else if (logType === 'requests') { + return [ + new LoggingWinston({ + ...gcsConfig, + logName: 'requests' + }) + ] + } + return [] + } +} diff --git a/packages/components/src/storage/IStorageProvider.ts b/packages/components/src/storage/IStorageProvider.ts new file mode 100644 index 000000000..cabdfa35c --- /dev/null +++ b/packages/components/src/storage/IStorageProvider.ts @@ -0,0 +1,98 @@ +import fs from 'node:fs' + +export interface FileInfo { + name: string + path: string + size: number +} + +export interface StorageResult { + path: string + totalSize: number +} + +export interface StorageSizeResult { + totalSize: number +} + +export interface IStorageProvider { + /** + * Get the storage type (e.g., 'local', 's3', 'gcs', 'azure') + */ + getStorageType(): string + + /** + * Get the provider configuration + */ + getConfig(): any + + /** + * Add files provided as base64 strings to storage + */ + addBase64FilesToStorage(fileBase64: string, chatflowid: string, fileNames: string[], orgId: string): Promise + + /** + * Add file buffer to storage as part of an array + */ + addArrayFilesToStorage(mime: string, bf: Buffer, fileName: string, fileNames: string[], ...paths: string[]): Promise + + /** + * Add a single file buffer to storage + */ + addSingleFileToStorage(mime: string, bf: Buffer, fileName: string, ...paths: string[]): Promise + + /** + * Retrieve a file from the upload directory + */ + getFileFromUpload(filePath: string): Promise + + /** + * Retrieve a file from generic storage + */ + getFileFromStorage(file: string, ...paths: string[]): Promise + + /** + * List files in a storage path + */ + getFilesListFromStorage(...paths: string[]): Promise + + /** + * Stream a file from storage + */ + streamStorageFile(chatflowId: string, chatId: string, fileName: string, orgId: string): Promise + + /** + * Remove multiple files from storage + */ + removeFilesFromStorage(...paths: string[]): Promise + + /** + * Remove a specific file from the upload directory + */ + removeSpecificFileFromUpload(filePath: string): Promise + + /** + * Remove a specific file from storage + */ + removeSpecificFileFromStorage(...paths: string[]): Promise + + /** + * Remove a complete folder and its contents from storage + */ + removeFolderFromStorage(...paths: string[]): Promise + + /** + * Calculate the total storage size for an organization + */ + getStorageSize(orgId: string): Promise + + /** + * Get the Multer storage engine for this provider + */ + getMulterStorage(): any + + /** + * Get the Winston logger transports for this provider + */ + getLoggerTransports(logType: 'server' | 'error' | 'requests', config?: any): any[] +} diff --git a/packages/components/src/storage/LocalStorageProvider.ts b/packages/components/src/storage/LocalStorageProvider.ts new file mode 100644 index 000000000..1d5f40557 --- /dev/null +++ b/packages/components/src/storage/LocalStorageProvider.ts @@ -0,0 +1,381 @@ +import path from 'node:path' +import fs from 'fs' +import multer from 'multer' +import DailyRotateFile from 'winston-daily-rotate-file' +import { transports } from 'winston' +import { BaseStorageProvider } from './BaseStorageProvider' +import { FileInfo, StorageResult, StorageSizeResult } from './IStorageProvider' + +export class LocalStorageProvider extends BaseStorageProvider { + constructor() { + super() + } + + getStorageType(): string { + return 'local' + } + + getConfig(): any { + return { + storagePath: this.storagePath + } + } + + async addBase64FilesToStorage(fileBase64: string, chatflowid: string, fileNames: string[], orgId: string): Promise { + // Validate chatflowid + this.validateChatflowId(chatflowid) + this.validatePathSecurity(chatflowid) + + const dir = this.buildPath(orgId, chatflowid) + if (!fs.existsSync(dir)) { + fs.mkdirSync(dir, { recursive: true }) + } + + const splitDataURI = fileBase64.split(',') + const filename = splitDataURI.pop()?.split(':')[1] ?? '' + const bf = Buffer.from(splitDataURI.pop() || '', 'base64') + const sanitizedFilename = this.sanitizeFilename(filename) + + const filePath = path.join(dir, sanitizedFilename) + fs.writeFileSync(filePath, bf) + fileNames.push(sanitizedFilename) + + const totalSize = await this.getStorageSize(orgId) + return { path: 'FILE-STORAGE::' + JSON.stringify(fileNames), totalSize: totalSize / 1024 / 1024 } + } + + async addArrayFilesToStorage( + mime: string, + bf: Buffer, + fileName: string, + fileNames: string[], + ...paths: string[] + ): Promise { + const dir = this.buildPath(...paths) + if (!fs.existsSync(dir)) { + fs.mkdirSync(dir, { recursive: true }) + } + const sanitizedFilename = this.sanitizeFilename(fileName) + const filePath = path.join(dir, sanitizedFilename) + fs.writeFileSync(filePath, bf) + fileNames.push(sanitizedFilename) + + const totalSize = await this.getStorageSize(paths[0]) + return { path: 'FILE-STORAGE::' + JSON.stringify(fileNames), totalSize: totalSize / 1024 / 1024 } + } + + async addSingleFileToStorage(mime: string, bf: Buffer, fileName: string, ...paths: string[]): Promise { + const dir = this.buildPath(...paths) + if (!fs.existsSync(dir)) { + fs.mkdirSync(dir, { recursive: true }) + } + const sanitizedFilename = this.sanitizeFilename(fileName) + const filePath = path.join(dir, sanitizedFilename) + fs.writeFileSync(filePath, bf) + + const totalSize = await this.getStorageSize(paths[0]) + return { path: 'FILE-STORAGE::' + sanitizedFilename, totalSize: totalSize / 1024 / 1024 } + } + + async getFileFromUpload(filePath: string): Promise { + return fs.readFileSync(filePath) + } + + async getFileFromStorage(file: string, ...paths: string[]): Promise { + const sanitizedFilename = this.sanitizeFilename(file) + const fileInStorage = this.buildPath(...paths.map((p) => this.sanitizeFilename(p)), sanitizedFilename) + try { + return fs.readFileSync(fileInStorage) + } catch (error) { + // Fallback: Check if file exists without the first path element (likely orgId) + if (paths.length > 1) { + const fallbackPaths = paths.slice(1) + const fallbackPath = this.buildPath(...fallbackPaths.map((p) => this.sanitizeFilename(p)), sanitizedFilename) + + if (fs.existsSync(fallbackPath)) { + // Create directory if it doesn't exist + const targetPath = fileInStorage + const dir = path.dirname(targetPath) + if (!fs.existsSync(dir)) { + fs.mkdirSync(dir, { recursive: true }) + } + + // Copy file to correct location with orgId + fs.copyFileSync(fallbackPath, targetPath) + + // Delete the old file + fs.unlinkSync(fallbackPath) + + // Clean up empty directories recursively + if (fallbackPaths.length > 0) { + this.cleanEmptyLocalFolders(this.buildPath(...fallbackPaths.map((p) => this.sanitizeFilename(p)).slice(0, -1))) + } + + return fs.readFileSync(targetPath) + } + } + throw error + } + } + + async streamStorageFile( + chatflowId: string, + chatId: string, + fileName: string, + orgId: string + ): Promise { + // Validate chatflowId and chatId + this.validateChatflowId(chatflowId) + this.validatePathSecurity(chatflowId, chatId) + + const sanitizedFilename = this.sanitizeFilename(fileName) + const filePath = this.buildPath(orgId, chatflowId, chatId, sanitizedFilename) + + //raise error if file path is not absolute + if (!path.isAbsolute(filePath)) throw new Error(`Invalid file path`) + //raise error if file path contains '..' + if (filePath.includes('..')) throw new Error(`Invalid file path`) + //only return from the storage folder + if (!filePath.startsWith(this.storagePath)) throw new Error(`Invalid file path`) + + if (fs.existsSync(filePath)) { + return fs.createReadStream(filePath) + } else { + // Fallback: Check if file exists without orgId + const fallbackPath = this.buildPath(chatflowId, chatId, sanitizedFilename) + + if (fs.existsSync(fallbackPath)) { + // Create directory if it doesn't exist + const dir = path.dirname(filePath) + if (!fs.existsSync(dir)) { + fs.mkdirSync(dir, { recursive: true }) + } + + // Copy file to correct location with orgId + fs.copyFileSync(fallbackPath, filePath) + + // Delete the old file + fs.unlinkSync(fallbackPath) + + // Clean up empty directories recursively + this.cleanEmptyLocalFolders(path.dirname(fallbackPath)) + + return fs.createReadStream(filePath) + } else { + throw new Error(`File ${fileName} not found`) + } + } + } + + async getFilesListFromStorage(...paths: string[]): Promise { + const directory = this.buildPath(...paths) + return this.getFilePaths(directory) + } + + private getFilePaths(dir: string): FileInfo[] { + let results: FileInfo[] = [] + + const readDirectory = (directory: string) => { + try { + if (!fs.existsSync(directory)) { + console.warn(`Directory does not exist: ${directory}`) + return + } + + const list = fs.readdirSync(directory) + list.forEach((file) => { + const filePath = path.join(directory, file) + try { + const stat = fs.statSync(filePath) + if (stat && stat.isDirectory()) { + readDirectory(filePath) + } else { + results.push({ name: file, path: filePath, size: stat.size }) + } + } catch (error) { + console.error(`Error processing file ${filePath}:`, error) + } + }) + } catch (error) { + console.error(`Error reading directory ${directory}:`, error) + } + } + + readDirectory(dir) + return results + } + + private cleanEmptyLocalFolders(dirPath: string) { + try { + // Stop if we reach the storage root + if (dirPath === this.storagePath) return + + // Check if directory exists + if (!fs.existsSync(dirPath)) return + + // Read directory contents + const files = fs.readdirSync(dirPath) + + // If directory is empty, delete it and check parent + if (files.length === 0) { + fs.rmdirSync(dirPath) + // Recursively check parent directory + this.cleanEmptyLocalFolders(path.dirname(dirPath)) + } + } catch (error) { + // Ignore errors during cleanup + console.error('Error cleaning empty folders:', error) + } + } + + async removeFilesFromStorage(...paths: string[]): Promise { + const directory = this.buildPath(...paths.map((p) => this.sanitizeFilename(p))) + await this.deleteLocalFolderRecursive(directory) + + const totalSize = await this.getStorageSize(paths[0]) + return { totalSize: totalSize / 1024 / 1024 } + } + + async removeSpecificFileFromUpload(filePath: string): Promise { + if (fs.existsSync(filePath)) { + fs.unlinkSync(filePath) + } + } + + async removeSpecificFileFromStorage(...paths: string[]): Promise { + const fileName = paths.pop() + if (fileName) { + const sanitizedFilename = this.sanitizeFilename(fileName) + paths.push(sanitizedFilename) + } + const file = this.buildPath(...paths.map((p) => this.sanitizeFilename(p))) + + // check if file exists, if not skip delete + const stat = fs.statSync(file, { throwIfNoEntry: false }) + if (stat && stat.isFile()) { + fs.unlinkSync(file) + } + + const totalSize = await this.getStorageSize(paths[0]) + return { totalSize: totalSize / 1024 / 1024 } + } + + async removeFolderFromStorage(...paths: string[]): Promise { + const directory = this.buildPath(...paths.map((p) => this.sanitizeFilename(p))) + await this.deleteLocalFolderRecursive(directory, true) + + const totalSize = await this.getStorageSize(paths[0]) + return { totalSize: totalSize / 1024 / 1024 } + } + + private async deleteLocalFolderRecursive(directory: string, deleteParentChatflowFolder?: boolean): Promise { + try { + // Check if the path exists + await fs.promises.access(directory) + + if (deleteParentChatflowFolder) { + await fs.promises.rm(directory, { recursive: true }) + return + } + + // Get stats of the path to determine if it's a file or directory + const stats = await fs.promises.stat(directory) + + if (stats.isDirectory()) { + // Read all directory contents + const files = await fs.promises.readdir(directory) + + // Recursively delete all contents + for (const file of files) { + const currentPath = path.join(directory, file) + await this.deleteLocalFolderRecursive(currentPath) + } + + // Delete the directory itself after emptying it + await fs.promises.rm(directory, { recursive: true }) + } else { + // If it's a file, delete it directly + await fs.promises.unlink(directory) + } + } catch (error) { + // Error handling - ignore if file/directory doesn't exist + } + } + + async getStorageSize(orgId: string): Promise { + if (!orgId) return 0 + const directory = this.buildPath(orgId) + return this.dirSize(directory) + } + + private async dirSize(directoryPath: string): Promise { + let totalSize = 0 + + const calculateSize = async (itemPath: string) => { + try { + const stats = await fs.promises.stat(itemPath) + + if (stats.isFile()) { + totalSize += stats.size + } else if (stats.isDirectory()) { + const files = await fs.promises.readdir(itemPath) + for (const file of files) { + await calculateSize(path.join(itemPath, file)) + } + } + } catch (error) { + // Ignore missing files/dirs during calculation + } + } + + await calculateSize(directoryPath) + return totalSize + } + + getMulterStorage(): multer.Multer { + const uploadPath = this.getUploadPath() + if (!fs.existsSync(uploadPath)) { + fs.mkdirSync(uploadPath, { recursive: true }) + } + return multer({ dest: uploadPath }) + } + + private getUploadPath(): string { + return process.env.BLOB_STORAGE_PATH + ? path.join(process.env.BLOB_STORAGE_PATH, 'uploads') + : path.join(this.getUserHome(), '.flowise', 'uploads') + } + + private getUserHome(): string { + return process.env.HOME || process.env.USERPROFILE || process.env.HOMEPATH || '' + } + + getLoggerTransports(logType: 'server' | 'error' | 'requests', config?: any): any[] { + const logDir = config?.logging?.dir || path.join(this.getUserHome(), '.flowise', 'logs') + + if (!fs.existsSync(logDir)) { + fs.mkdirSync(logDir, { recursive: true }) + } + + if (logType === 'server') { + return [ + new DailyRotateFile({ + filename: path.join(logDir, config?.logging?.server?.filename ?? 'server-%DATE%.log'), + datePattern: 'YYYY-MM-DD-HH', + maxSize: '20m', + level: config?.logging?.server?.level ?? 'info' + }) + ] + } else if (logType === 'requests') { + return [ + new transports.File({ + filename: path.join(logDir, config?.logging?.express?.filename ?? 'server-requests.log.jsonl'), + level: config?.logging?.express?.level ?? 'debug' + }) + ] + } + + // For 'error' type, return empty array (handled by exceptionHandlers in logger.ts) + return [] + } +} diff --git a/packages/components/src/storage/S3StorageProvider.ts b/packages/components/src/storage/S3StorageProvider.ts new file mode 100644 index 000000000..dec9d99d4 --- /dev/null +++ b/packages/components/src/storage/S3StorageProvider.ts @@ -0,0 +1,538 @@ +import { + DeleteObjectsCommand, + GetObjectCommand, + ListObjectsCommand, + ListObjectsV2Command, + PutObjectCommand, + S3Client, + S3ClientConfig +} from '@aws-sdk/client-s3' +import { Readable } from 'node:stream' +import multer from 'multer' +import multerS3 from 'multer-s3' +import { transports } from 'winston' +import { v4 as uuidv4 } from 'uuid' +import { BaseStorageProvider } from './BaseStorageProvider' +import { FileInfo, StorageResult, StorageSizeResult } from './IStorageProvider' + +const { S3StreamLogger } = require('s3-streamlogger') + +export class S3StorageProvider extends BaseStorageProvider { + private s3Client: S3Client + private bucket: string + private s3Config: S3ClientConfig + + constructor() { + super() + const config = this.initS3Config() + this.s3Client = config.s3Client + this.bucket = config.bucket + this.s3Config = config.s3Config + } + + private initS3Config(): { s3Client: S3Client; bucket: string; s3Config: S3ClientConfig } { + const accessKeyId = process.env.S3_STORAGE_ACCESS_KEY_ID + const secretAccessKey = process.env.S3_STORAGE_SECRET_ACCESS_KEY + const region = process.env.S3_STORAGE_REGION + const bucket = process.env.S3_STORAGE_BUCKET_NAME + const customURL = process.env.S3_ENDPOINT_URL + const forcePathStyle = process.env.S3_FORCE_PATH_STYLE === 'true' + + if (!region || region.trim() === '' || !bucket || bucket.trim() === '') { + throw new Error('S3 storage configuration is missing') + } + + const s3Config: S3ClientConfig = { + region: region, + forcePathStyle: forcePathStyle + } + + // Only include endpoint if customURL is not empty + if (customURL && customURL.trim() !== '') { + s3Config.endpoint = customURL + } + + if (accessKeyId && accessKeyId.trim() !== '' && secretAccessKey && secretAccessKey.trim() !== '') { + s3Config.credentials = { + accessKeyId: accessKeyId, + secretAccessKey: secretAccessKey + } + } + + const s3Client = new S3Client(s3Config) + return { s3Client, bucket, s3Config } + } + + getStorageType(): string { + return 's3' + } + + getConfig(): any { + return { + bucket: this.bucket, + region: process.env.S3_STORAGE_REGION, + endpoint: process.env.S3_ENDPOINT_URL + } + } + + async addBase64FilesToStorage(fileBase64: string, chatflowid: string, fileNames: string[], orgId: string): Promise { + // Validate chatflowid + this.validateChatflowId(chatflowid) + this.validatePathSecurity(chatflowid) + + const splitDataURI = fileBase64.split(',') + const filename = splitDataURI.pop()?.split(':')[1] ?? '' + const bf = Buffer.from(splitDataURI.pop() || '', 'base64') + const mime = splitDataURI[0].split(':')[1].split(';')[0] + + const sanitizedFilename = this.sanitizeFilename(filename) + const Key = orgId + '/' + chatflowid + '/' + sanitizedFilename + + const putObjCmd = new PutObjectCommand({ + Bucket: this.bucket, + Key, + ContentEncoding: 'base64', + ContentType: mime, + Body: bf + }) + await this.s3Client.send(putObjCmd) + + fileNames.push(sanitizedFilename) + const totalSize = await this.getStorageSize(orgId) + + return { path: 'FILE-STORAGE::' + JSON.stringify(fileNames), totalSize: totalSize / 1024 / 1024 } + } + + async addArrayFilesToStorage( + mime: string, + bf: Buffer, + fileName: string, + fileNames: string[], + ...paths: string[] + ): Promise { + const sanitizedFilename = this.sanitizeFilename(fileName) + + let Key = paths.reduce((acc, cur) => acc + '/' + cur, '') + '/' + sanitizedFilename + if (Key.startsWith('/')) { + Key = Key.substring(1) + } + + const putObjCmd = new PutObjectCommand({ + Bucket: this.bucket, + Key, + ContentEncoding: 'base64', + ContentType: mime, + Body: bf + }) + await this.s3Client.send(putObjCmd) + fileNames.push(sanitizedFilename) + + const totalSize = await this.getStorageSize(paths[0]) + return { path: 'FILE-STORAGE::' + JSON.stringify(fileNames), totalSize: totalSize / 1024 / 1024 } + } + + async addSingleFileToStorage(mime: string, bf: Buffer, fileName: string, ...paths: string[]): Promise { + const sanitizedFilename = this.sanitizeFilename(fileName) + + let Key = paths.reduce((acc, cur) => acc + '/' + cur, '') + '/' + sanitizedFilename + if (Key.startsWith('/')) { + Key = Key.substring(1) + } + + const putObjCmd = new PutObjectCommand({ + Bucket: this.bucket, + Key, + ContentEncoding: 'base64', + ContentType: mime, + Body: bf + }) + await this.s3Client.send(putObjCmd) + + const totalSize = await this.getStorageSize(paths[0]) + return { path: 'FILE-STORAGE::' + sanitizedFilename, totalSize: totalSize / 1024 / 1024 } + } + + async getFileFromUpload(filePath: string): Promise { + // For S3, the filePath is the S3 key + let Key = filePath + if (Key.startsWith('/')) { + Key = Key.substring(1) + } + + const getParams = { + Bucket: this.bucket, + Key + } + + const response = await this.s3Client.send(new GetObjectCommand(getParams)) + const body = response.Body + if (body instanceof Readable) { + const streamToString = await body.transformToString('base64') + if (streamToString) { + return Buffer.from(streamToString, 'base64') + } + } + const byteArray = await response.Body!.transformToByteArray() + return Buffer.from(byteArray) + } + + async getFileFromStorage(file: string, ...paths: string[]): Promise { + const sanitizedFilename = this.sanitizeFilename(file) + + let Key = paths.reduce((acc, cur) => acc + '/' + cur, '') + '/' + sanitizedFilename + if (Key.startsWith('/')) { + Key = Key.substring(1) + } + + try { + const getParams = { + Bucket: this.bucket, + Key + } + + const response = await this.s3Client.send(new GetObjectCommand(getParams)) + const body = response.Body + if (body instanceof Readable) { + const streamToString = await body.transformToString('base64') + if (streamToString) { + return Buffer.from(streamToString, 'base64') + } + } + const byteArray = await response.Body!.transformToByteArray() + return Buffer.from(byteArray) + } catch (error) { + // Fallback: Check if file exists without the first path element (likely orgId) + if (paths.length > 1) { + const fallbackPaths = paths.slice(1) + let fallbackKey = fallbackPaths.reduce((acc, cur) => acc + '/' + cur, '') + '/' + sanitizedFilename + if (fallbackKey.startsWith('/')) { + fallbackKey = fallbackKey.substring(1) + } + + try { + const fallbackParams = { + Bucket: this.bucket, + Key: fallbackKey + } + const fallbackResponse = await this.s3Client.send(new GetObjectCommand(fallbackParams)) + const fallbackBody = fallbackResponse.Body + + // Get the file content + let fileContent: Buffer + if (fallbackBody instanceof Readable) { + const streamToString = await fallbackBody.transformToString('base64') + if (streamToString) { + fileContent = Buffer.from(streamToString, 'base64') + } else { + const byteArray = await fallbackBody.transformToByteArray() + fileContent = Buffer.from(byteArray) + } + } else { + const byteArray = await fallbackBody!.transformToByteArray() + fileContent = Buffer.from(byteArray) + } + + // Move to correct location with orgId + const putObjCmd = new PutObjectCommand({ + Bucket: this.bucket, + Key, + Body: fileContent + }) + await this.s3Client.send(putObjCmd) + + // Delete the old file + await this.s3Client.send( + new DeleteObjectsCommand({ + Bucket: this.bucket, + Delete: { + Objects: [{ Key: fallbackKey }], + Quiet: false + } + }) + ) + + // Check if the directory is empty and delete recursively if needed + if (fallbackPaths.length > 0) { + await this.cleanEmptyS3Folders(fallbackPaths[0]) + } + + return fileContent + } catch (fallbackError) { + throw error + } + } else { + throw error + } + } + } + + async streamStorageFile(chatflowId: string, chatId: string, fileName: string, orgId: string): Promise { + // Validate chatflowId and chatId + this.validateChatflowId(chatflowId) + this.validatePathSecurity(chatflowId, chatId) + + const sanitizedFilename = this.sanitizeFilename(fileName) + const Key = orgId + '/' + chatflowId + '/' + chatId + '/' + sanitizedFilename + + const getParams = { + Bucket: this.bucket, + Key + } + + try { + const response = await this.s3Client.send(new GetObjectCommand(getParams)) + const body = response.Body + if (body instanceof Readable) { + const blob = await body.transformToByteArray() + return Buffer.from(blob) + } + } catch (error) { + // Fallback: Check if file exists without orgId + const fallbackKey = chatflowId + '/' + chatId + '/' + sanitizedFilename + try { + const fallbackParams = { + Bucket: this.bucket, + Key: fallbackKey + } + const fallbackResponse = await this.s3Client.send(new GetObjectCommand(fallbackParams)) + const fallbackBody = fallbackResponse.Body + + // If found, copy to correct location with orgId + if (fallbackBody) { + let fileContent: Buffer + if (fallbackBody instanceof Readable) { + const blob = await fallbackBody.transformToByteArray() + fileContent = Buffer.from(blob) + } else { + const byteArray = await fallbackBody.transformToByteArray() + fileContent = Buffer.from(byteArray) + } + + // Move to correct location with orgId + const putObjCmd = new PutObjectCommand({ + Bucket: this.bucket, + Key, + Body: fileContent + }) + await this.s3Client.send(putObjCmd) + + // Delete the old file + await this.s3Client.send( + new DeleteObjectsCommand({ + Bucket: this.bucket, + Delete: { + Objects: [{ Key: fallbackKey }], + Quiet: false + } + }) + ) + + // Check if the directory is empty and delete recursively if needed + await this.cleanEmptyS3Folders(chatflowId) + + return fileContent + } + } catch (fallbackError) { + throw new Error(`File ${fileName} not found`) + } + } + } + + async getFilesListFromStorage(...paths: string[]): Promise { + let Key = paths.reduce((acc, cur) => acc + '/' + cur, '') + if (Key.startsWith('/')) { + Key = Key.substring(1) + } + + const listCommand = new ListObjectsV2Command({ + Bucket: this.bucket, + Prefix: Key + }) + const list = await this.s3Client.send(listCommand) + + if (list.Contents && list.Contents.length > 0) { + return list.Contents.map((item) => ({ + name: item.Key?.split('/').pop() || '', + path: item.Key ?? '', + size: item.Size || 0 + })) + } else { + return [] + } + } + + async removeFilesFromStorage(...paths: string[]): Promise { + let Key = paths.reduce((acc, cur) => acc + '/' + cur, '') + if (Key.startsWith('/')) { + Key = Key.substring(1) + } + + await this.deleteS3Folder(Key) + + const totalSize = await this.getStorageSize(paths[0]) + return { totalSize: totalSize / 1024 / 1024 } + } + + async removeSpecificFileFromUpload(filePath: string): Promise { + let Key = filePath + if (Key.startsWith('/')) { + Key = Key.substring(1) + } + await this.deleteS3Folder(Key) + } + + async removeSpecificFileFromStorage(...paths: string[]): Promise { + let Key = paths.reduce((acc, cur) => acc + '/' + cur, '') + if (Key.startsWith('/')) { + Key = Key.substring(1) + } + await this.deleteS3Folder(Key) + + const totalSize = await this.getStorageSize(paths[0]) + return { totalSize: totalSize / 1024 / 1024 } + } + + async removeFolderFromStorage(...paths: string[]): Promise { + let Key = paths.reduce((acc, cur) => acc + '/' + cur, '') + if (Key.startsWith('/')) { + Key = Key.substring(1) + } + await this.deleteS3Folder(Key) + + const totalSize = await this.getStorageSize(paths[0]) + return { totalSize: totalSize / 1024 / 1024 } + } + + private async deleteS3Folder(location: string): Promise { + let count = 0 + + const recursiveS3Delete = async (token?: string): Promise => { + const listCommand = new ListObjectsV2Command({ + Bucket: this.bucket, + Prefix: location, + ContinuationToken: token + }) + const list = await this.s3Client.send(listCommand) + + if (list.KeyCount) { + const deleteCommand = new DeleteObjectsCommand({ + Bucket: this.bucket, + Delete: { + Objects: list.Contents?.map((item) => ({ Key: item.Key })), + Quiet: false + } + }) + const deleted = await this.s3Client.send(deleteCommand) + // @ts-ignore + count += deleted.Deleted?.length || 0 + + if (deleted.Errors) { + deleted.Errors.map((error: any) => console.error(`${error.Key} could not be deleted - ${error.Code}`)) + } + } + + if (list.NextContinuationToken) { + return recursiveS3Delete(list.NextContinuationToken) + } + + return `${count} files deleted from S3` + } + + return recursiveS3Delete() + } + + private async cleanEmptyS3Folders(prefix: string): Promise { + try { + if (!prefix) return + + const listCmd = new ListObjectsV2Command({ + Bucket: this.bucket, + Prefix: prefix + '/', + Delimiter: '/' + }) + + const response = await this.s3Client.send(listCmd) + + if ( + (response.Contents?.length === 0 || !response.Contents) && + (response.CommonPrefixes?.length === 0 || !response.CommonPrefixes) + ) { + await this.s3Client.send( + new DeleteObjectsCommand({ + Bucket: this.bucket, + Delete: { + Objects: [{ Key: prefix + '/' }], + Quiet: true + } + }) + ) + + const parentPrefix = prefix.substring(0, prefix.lastIndexOf('/')) + if (parentPrefix) { + await this.cleanEmptyS3Folders(parentPrefix) + } + } + } catch (error) { + console.error('Error cleaning empty S3 folders:', error) + } + } + + async getStorageSize(orgId: string): Promise { + if (!orgId) return 0 + + const getCmd = new ListObjectsCommand({ + Bucket: this.bucket, + Prefix: orgId + }) + const headObj = await this.s3Client.send(getCmd) + let totalSize = 0 + for (const obj of headObj.Contents || []) { + totalSize += obj.Size || 0 + } + return totalSize + } + + getMulterStorage(): multer.Multer { + return multer({ + storage: multerS3({ + s3: this.s3Client, + bucket: this.bucket, + metadata: function (req: any, file: any, cb: any) { + cb(null, { fieldName: file.fieldname, originalName: file.originalname }) + }, + key: function (req: any, file: any, cb: any) { + cb(null, `${uuidv4()}`) + } + }) + }) + } + + getLoggerTransports(logType: 'server' | 'error' | 'requests'): any[] { + if (logType === 'server') { + const s3ServerStream = new S3StreamLogger({ + bucket: this.bucket, + folder: 'logs/server', + name_format: `server-%Y-%m-%d-%H-%M-%S-%L.log`, + config: this.s3Config + }) + return [new transports.Stream({ stream: s3ServerStream })] + } else if (logType === 'error') { + const s3ErrorStream = new S3StreamLogger({ + bucket: this.bucket, + folder: 'logs/error', + name_format: `server-error-%Y-%m-%d-%H-%M-%S-%L.log`, + config: this.s3Config + }) + return [new transports.Stream({ stream: s3ErrorStream })] + } else if (logType === 'requests') { + const s3ServerReqStream = new S3StreamLogger({ + bucket: this.bucket, + folder: 'logs/requests', + name_format: `server-requests-%Y-%m-%d-%H-%M-%S-%L.log.jsonl`, + config: this.s3Config + }) + return [new transports.Stream({ stream: s3ServerReqStream })] + } + return [] + } +} diff --git a/packages/components/src/storage/StorageProviderFactory.ts b/packages/components/src/storage/StorageProviderFactory.ts new file mode 100644 index 000000000..06d47e4d6 --- /dev/null +++ b/packages/components/src/storage/StorageProviderFactory.ts @@ -0,0 +1,40 @@ +import { IStorageProvider } from './IStorageProvider' +import { LocalStorageProvider } from './LocalStorageProvider' +import { S3StorageProvider } from './S3StorageProvider' +import { GCSStorageProvider } from './GCSStorageProvider' +import { AzureBlobStorageProvider } from './AzureBlobStorageProvider' + +/** + * Factory for creating and managing storage provider instances. + * Uses singleton pattern to ensure only one provider instance exists. + */ +export class StorageProviderFactory { + private static instance: IStorageProvider | null = null + + /** + * Get the storage provider instance based on STORAGE_TYPE environment variable. + * Creates a new instance if one doesn't exist. + */ + static getProvider(): IStorageProvider { + if (!StorageProviderFactory.instance) { + const storageType = process.env.STORAGE_TYPE || 'local' + + switch (storageType) { + case 's3': + StorageProviderFactory.instance = new S3StorageProvider() + break + case 'gcs': + StorageProviderFactory.instance = new GCSStorageProvider() + break + case 'azure': + StorageProviderFactory.instance = new AzureBlobStorageProvider() + break + case 'local': + default: + StorageProviderFactory.instance = new LocalStorageProvider() + break + } + } + return StorageProviderFactory.instance + } +} diff --git a/packages/components/src/storage/index.ts b/packages/components/src/storage/index.ts new file mode 100644 index 000000000..3527e5286 --- /dev/null +++ b/packages/components/src/storage/index.ts @@ -0,0 +1,14 @@ +// Interfaces and types +export * from './IStorageProvider' + +// Base provider +export { BaseStorageProvider } from './BaseStorageProvider' + +// Provider implementations +export { LocalStorageProvider } from './LocalStorageProvider' +export { S3StorageProvider } from './S3StorageProvider' +export { GCSStorageProvider } from './GCSStorageProvider' +export { AzureBlobStorageProvider } from './AzureBlobStorageProvider' + +// Factory +export { StorageProviderFactory } from './StorageProviderFactory' diff --git a/packages/components/src/storageUtils.ts b/packages/components/src/storageUtils.ts index 66cdb5dc0..eeff63e9c 100644 --- a/packages/components/src/storageUtils.ts +++ b/packages/components/src/storageUtils.ts @@ -1,121 +1,16 @@ -import { - DeleteObjectsCommand, - GetObjectCommand, - ListObjectsCommand, - ListObjectsV2Command, - PutObjectCommand, - S3Client, - S3ClientConfig -} from '@aws-sdk/client-s3' -import { Storage } from '@google-cloud/storage' import fs from 'fs' -import { Readable } from 'node:stream' import path from 'path' -import sanitize from 'sanitize-filename' import { getUserHome } from './utils' -import { isPathTraversal, isUnsafeFilePath, isValidUUID } from './validator' - -const dirSize = async (directoryPath: string) => { - let totalSize = 0 - - async function calculateSize(itemPath: string) { - const stats = await fs.promises.stat(itemPath) - - if (stats.isFile()) { - totalSize += stats.size - } else if (stats.isDirectory()) { - const files = await fs.promises.readdir(itemPath) - for (const file of files) { - await calculateSize(path.join(itemPath, file)) - } - } - } - - await calculateSize(directoryPath) - return totalSize -} +import { StorageProviderFactory, StorageResult, StorageSizeResult, FileInfo } from './storage' export const addBase64FilesToStorage = async ( fileBase64: string, chatflowid: string, fileNames: string[], orgId: string -): Promise<{ path: string; totalSize: number }> => { - // Validate chatflowid - if (!chatflowid || !isValidUUID(chatflowid)) { - throw new Error('Invalid chatflowId format - must be a valid UUID') - } - - // Check for path traversal attempts - if (isPathTraversal(chatflowid)) { - throw new Error('Invalid path characters detected in chatflowId') - } - - const storageType = getStorageType() - if (storageType === 's3') { - const { s3Client, Bucket } = getS3Config() - - const splitDataURI = fileBase64.split(',') - const filename = splitDataURI.pop()?.split(':')[1] ?? '' - const bf = Buffer.from(splitDataURI.pop() || '', 'base64') - const mime = splitDataURI[0].split(':')[1].split(';')[0] - - const sanitizedFilename = _sanitizeFilename(filename) - const Key = orgId + '/' + chatflowid + '/' + sanitizedFilename - - const putObjCmd = new PutObjectCommand({ - Bucket, - Key, - ContentEncoding: 'base64', // required for binary data - ContentType: mime, - Body: bf - }) - await s3Client.send(putObjCmd) - - fileNames.push(sanitizedFilename) - const totalSize = await getS3StorageSize(orgId) - - return { path: 'FILE-STORAGE::' + JSON.stringify(fileNames), totalSize: totalSize / 1024 / 1024 } - } else if (storageType === 'gcs') { - const { bucket } = getGcsClient() - const splitDataURI = fileBase64.split(',') - const filename = splitDataURI.pop()?.split(':')[1] ?? '' - const bf = Buffer.from(splitDataURI.pop() || '', 'base64') - const mime = splitDataURI[0].split(':')[1].split(';')[0] - const sanitizedFilename = _sanitizeFilename(filename) - const normalizedChatflowid = chatflowid.replace(/\\/g, '/') - const normalizedFilename = sanitizedFilename.replace(/\\/g, '/') - const filePath = `${normalizedChatflowid}/${normalizedFilename}` - const file = bucket.file(filePath) - await new Promise((resolve, reject) => { - file.createWriteStream({ contentType: mime, metadata: { contentEncoding: 'base64' } }) - .on('error', (err) => reject(err)) - .on('finish', () => resolve()) - .end(bf) - }) - fileNames.push(sanitizedFilename) - const totalSize = await getGCSStorageSize(orgId) - - return { path: 'FILE-STORAGE::' + JSON.stringify(fileNames), totalSize: totalSize / 1024 / 1024 } - } else { - const dir = path.join(getStoragePath(), orgId, chatflowid) - if (!fs.existsSync(dir)) { - fs.mkdirSync(dir, { recursive: true }) - } - - const splitDataURI = fileBase64.split(',') - const filename = splitDataURI.pop()?.split(':')[1] ?? '' - const bf = Buffer.from(splitDataURI.pop() || '', 'base64') - const sanitizedFilename = _sanitizeFilename(filename) - - const filePath = path.join(dir, sanitizedFilename) - - fs.writeFileSync(filePath, bf) - fileNames.push(sanitizedFilename) - - const totalSize = await dirSize(path.join(getStoragePath(), orgId)) - return { path: 'FILE-STORAGE::' + JSON.stringify(fileNames), totalSize: totalSize / 1024 / 1024 } - } +): Promise => { + const provider = StorageProviderFactory.getProvider() + return provider.addBase64FilesToStorage(fileBase64, chatflowid, fileNames, orgId) } export const addArrayFilesToStorage = async ( @@ -124,407 +19,29 @@ export const addArrayFilesToStorage = async ( fileName: string, fileNames: string[], ...paths: string[] -): Promise<{ path: string; totalSize: number }> => { - const storageType = getStorageType() - - const sanitizedFilename = _sanitizeFilename(fileName) - if (storageType === 's3') { - const { s3Client, Bucket } = getS3Config() - - let Key = paths.reduce((acc, cur) => acc + '/' + cur, '') + '/' + sanitizedFilename - if (Key.startsWith('/')) { - Key = Key.substring(1) - } - - const putObjCmd = new PutObjectCommand({ - Bucket, - Key, - ContentEncoding: 'base64', // required for binary data - ContentType: mime, - Body: bf - }) - await s3Client.send(putObjCmd) - fileNames.push(sanitizedFilename) - - const totalSize = await getS3StorageSize(paths[0]) - - return { path: 'FILE-STORAGE::' + JSON.stringify(fileNames), totalSize: totalSize / 1024 / 1024 } - } else if (storageType === 'gcs') { - const { bucket } = getGcsClient() - const normalizedPaths = paths.map((p) => p.replace(/\\/g, '/')) - const normalizedFilename = sanitizedFilename.replace(/\\/g, '/') - const filePath = [...normalizedPaths, normalizedFilename].join('/') - const file = bucket.file(filePath) - await new Promise((resolve, reject) => { - file.createWriteStream() - .on('error', (err) => reject(err)) - .on('finish', () => resolve()) - .end(bf) - }) - fileNames.push(sanitizedFilename) - - const totalSize = await getGCSStorageSize(paths[0]) - - return { path: 'FILE-STORAGE::' + JSON.stringify(fileNames), totalSize: totalSize / 1024 / 1024 } - } else { - const dir = path.join(getStoragePath(), ...paths.map(_sanitizeFilename)) - if (!fs.existsSync(dir)) { - fs.mkdirSync(dir, { recursive: true }) - } - const filePath = path.join(dir, sanitizedFilename) - fs.writeFileSync(filePath, bf) - fileNames.push(sanitizedFilename) - - const totalSize = await dirSize(path.join(getStoragePath(), paths[0])) - - return { path: 'FILE-STORAGE::' + JSON.stringify(fileNames), totalSize: totalSize / 1024 / 1024 } - } +): Promise => { + const provider = StorageProviderFactory.getProvider() + return provider.addArrayFilesToStorage(mime, bf, fileName, fileNames, ...paths) } -export const addSingleFileToStorage = async ( - mime: string, - bf: Buffer, - fileName: string, - ...paths: string[] -): Promise<{ path: string; totalSize: number }> => { - const storageType = getStorageType() - const sanitizedFilename = _sanitizeFilename(fileName) - - if (storageType === 's3') { - const { s3Client, Bucket } = getS3Config() - - let Key = paths.reduce((acc, cur) => acc + '/' + cur, '') + '/' + sanitizedFilename - if (Key.startsWith('/')) { - Key = Key.substring(1) - } - - const putObjCmd = new PutObjectCommand({ - Bucket, - Key, - ContentEncoding: 'base64', // required for binary data - ContentType: mime, - Body: bf - }) - await s3Client.send(putObjCmd) - - const totalSize = await getS3StorageSize(paths[0]) - - return { path: 'FILE-STORAGE::' + sanitizedFilename, totalSize: totalSize / 1024 / 1024 } - } else if (storageType === 'gcs') { - const { bucket } = getGcsClient() - const normalizedPaths = paths.map((p) => p.replace(/\\/g, '/')) - const normalizedFilename = sanitizedFilename.replace(/\\/g, '/') - const filePath = [...normalizedPaths, normalizedFilename].join('/') - const file = bucket.file(filePath) - await new Promise((resolve, reject) => { - file.createWriteStream({ contentType: mime, metadata: { contentEncoding: 'base64' } }) - .on('error', (err) => reject(err)) - .on('finish', () => resolve()) - .end(bf) - }) - - const totalSize = await getGCSStorageSize(paths[0]) - - return { path: 'FILE-STORAGE::' + sanitizedFilename, totalSize: totalSize / 1024 / 1024 } - } else { - const dir = path.join(getStoragePath(), ...paths.map(_sanitizeFilename)) - if (!fs.existsSync(dir)) { - fs.mkdirSync(dir, { recursive: true }) - } - const filePath = path.join(dir, sanitizedFilename) - fs.writeFileSync(filePath, bf) - - const totalSize = await dirSize(path.join(getStoragePath(), paths[0])) - return { path: 'FILE-STORAGE::' + sanitizedFilename, totalSize: totalSize / 1024 / 1024 } - } +export const addSingleFileToStorage = async (mime: string, bf: Buffer, fileName: string, ...paths: string[]): Promise => { + const provider = StorageProviderFactory.getProvider() + return provider.addSingleFileToStorage(mime, bf, fileName, ...paths) } export const getFileFromUpload = async (filePath: string): Promise => { - const storageType = getStorageType() - if (storageType === 's3') { - const { s3Client, Bucket } = getS3Config() - - let Key = filePath - // remove the first '/' if it exists - if (Key.startsWith('/')) { - Key = Key.substring(1) - } - const getParams = { - Bucket, - Key - } - - const response = await s3Client.send(new GetObjectCommand(getParams)) - const body = response.Body - if (body instanceof Readable) { - const streamToString = await body.transformToString('base64') - if (streamToString) { - return Buffer.from(streamToString, 'base64') - } - } - // @ts-ignore - const buffer = Buffer.concat(response.Body.toArray()) - return buffer - } else if (storageType === 'gcs') { - const { bucket } = getGcsClient() - const file = bucket.file(filePath) - const [buffer] = await file.download() - return buffer - } else { - return fs.readFileSync(filePath) - } + const provider = StorageProviderFactory.getProvider() + return provider.getFileFromUpload(filePath) } export const getFileFromStorage = async (file: string, ...paths: string[]): Promise => { - const storageType = getStorageType() - const sanitizedFilename = _sanitizeFilename(file) - - if (storageType === 's3') { - const { s3Client, Bucket } = getS3Config() - - let Key = paths.reduce((acc, cur) => acc + '/' + cur, '') + '/' + sanitizedFilename - if (Key.startsWith('/')) { - Key = Key.substring(1) - } - - try { - const getParams = { - Bucket, - Key - } - - const response = await s3Client.send(new GetObjectCommand(getParams)) - const body = response.Body - if (body instanceof Readable) { - const streamToString = await body.transformToString('base64') - if (streamToString) { - return Buffer.from(streamToString, 'base64') - } - } - // @ts-ignore - const buffer = Buffer.concat(response.Body.toArray()) - return buffer - } catch (error) { - // Fallback: Check if file exists without the first path element (likely orgId) - if (paths.length > 1) { - const fallbackPaths = paths.slice(1) - let fallbackKey = fallbackPaths.reduce((acc, cur) => acc + '/' + cur, '') + '/' + sanitizedFilename - if (fallbackKey.startsWith('/')) { - fallbackKey = fallbackKey.substring(1) - } - - try { - const fallbackParams = { - Bucket, - Key: fallbackKey - } - const fallbackResponse = await s3Client.send(new GetObjectCommand(fallbackParams)) - const fallbackBody = fallbackResponse.Body - - // Get the file content - let fileContent: Buffer - if (fallbackBody instanceof Readable) { - const streamToString = await fallbackBody.transformToString('base64') - if (streamToString) { - fileContent = Buffer.from(streamToString, 'base64') - } else { - // @ts-ignore - fileContent = Buffer.concat(fallbackBody.toArray()) - } - } else { - // @ts-ignore - fileContent = Buffer.concat(fallbackBody.toArray()) - } - - // Move to correct location with orgId - const putObjCmd = new PutObjectCommand({ - Bucket, - Key, - Body: fileContent - }) - await s3Client.send(putObjCmd) - - // Delete the old file - await s3Client.send( - new DeleteObjectsCommand({ - Bucket, - Delete: { - Objects: [{ Key: fallbackKey }], - Quiet: false - } - }) - ) - - // Check if the directory is empty and delete recursively if needed - if (fallbackPaths.length > 0) { - await _cleanEmptyS3Folders(s3Client, Bucket, fallbackPaths[0]) - } - - return fileContent - } catch (fallbackError) { - // Throw the original error since the fallback also failed - throw error - } - } else { - throw error - } - } - } else if (storageType === 'gcs') { - const { bucket } = getGcsClient() - const normalizedPaths = paths.map((p) => p.replace(/\\/g, '/')) - const normalizedFilename = sanitizedFilename.replace(/\\/g, '/') - const filePath = [...normalizedPaths, normalizedFilename].join('/') - - try { - const file = bucket.file(filePath) - const [buffer] = await file.download() - return buffer - } catch (error) { - // Fallback: Check if file exists without the first path element (likely orgId) - if (normalizedPaths.length > 1) { - const fallbackPaths = normalizedPaths.slice(1) - const fallbackPath = [...fallbackPaths, normalizedFilename].join('/') - - try { - const fallbackFile = bucket.file(fallbackPath) - const [buffer] = await fallbackFile.download() - - // Move to correct location with orgId - const file = bucket.file(filePath) - await new Promise((resolve, reject) => { - file.createWriteStream() - .on('error', (err) => reject(err)) - .on('finish', () => resolve()) - .end(buffer) - }) - - // Delete the old file - await fallbackFile.delete() - - // Check if the directory is empty and delete recursively if needed - if (fallbackPaths.length > 0) { - await _cleanEmptyGCSFolders(bucket, fallbackPaths[0]) - } - - return buffer - } catch (fallbackError) { - // Throw the original error since the fallback also failed - throw error - } - } else { - throw error - } - } - } else { - try { - const fileInStorage = path.join(getStoragePath(), ...paths.map(_sanitizeFilename), sanitizedFilename) - return fs.readFileSync(fileInStorage) - } catch (error) { - // Fallback: Check if file exists without the first path element (likely orgId) - if (paths.length > 1) { - const fallbackPaths = paths.slice(1) - const fallbackPath = path.join(getStoragePath(), ...fallbackPaths.map(_sanitizeFilename), sanitizedFilename) - - if (fs.existsSync(fallbackPath)) { - // Create directory if it doesn't exist - const targetPath = path.join(getStoragePath(), ...paths.map(_sanitizeFilename), sanitizedFilename) - const dir = path.dirname(targetPath) - if (!fs.existsSync(dir)) { - fs.mkdirSync(dir, { recursive: true }) - } - - // Copy file to correct location with orgId - fs.copyFileSync(fallbackPath, targetPath) - - // Delete the old file - fs.unlinkSync(fallbackPath) - - // Clean up empty directories recursively - if (fallbackPaths.length > 0) { - _cleanEmptyLocalFolders(path.join(getStoragePath(), ...fallbackPaths.map(_sanitizeFilename).slice(0, -1))) - } - - return fs.readFileSync(targetPath) - } else { - throw error - } - } else { - throw error - } - } - } + const provider = StorageProviderFactory.getProvider() + return provider.getFileFromStorage(file, ...paths) } -export const getFilesListFromStorage = async (...paths: string[]): Promise> => { - const storageType = getStorageType() - if (storageType === 's3') { - const { s3Client, Bucket } = getS3Config() - - let Key = paths.reduce((acc, cur) => acc + '/' + cur, '') - if (Key.startsWith('/')) { - Key = Key.substring(1) - } - - const listCommand = new ListObjectsV2Command({ - Bucket, - Prefix: Key - }) - const list = await s3Client.send(listCommand) - - if (list.Contents && list.Contents.length > 0) { - return list.Contents.map((item) => ({ - name: item.Key?.split('/').pop() || '', - path: item.Key ?? '', - size: item.Size || 0 - })) - } else { - return [] - } - } else { - const directory = path.join(getStoragePath(), ...paths) - const filesList = getFilePaths(directory) - return filesList - } -} - -interface FileInfo { - name: string - path: string - size: number -} - -function getFilePaths(dir: string): FileInfo[] { - let results: FileInfo[] = [] - - function readDirectory(directory: string) { - try { - if (!fs.existsSync(directory)) { - console.warn(`Directory does not exist: ${directory}`) - return - } - - const list = fs.readdirSync(directory) - list.forEach((file) => { - const filePath = path.join(directory, file) - try { - const stat = fs.statSync(filePath) - if (stat && stat.isDirectory()) { - readDirectory(filePath) - } else { - const sizeInMB = stat.size / (1024 * 1024) - results.push({ name: file, path: filePath, size: sizeInMB }) - } - } catch (error) { - console.error(`Error processing file ${filePath}:`, error) - } - }) - } catch (error) { - console.error(`Error reading directory ${directory}:`, error) - } - } - - readDirectory(dir) - return results +export const getFilesListFromStorage = async (...paths: string[]): Promise => { + const provider = StorageProviderFactory.getProvider() + return provider.getFilesListFromStorage(...paths) } /** @@ -541,204 +58,30 @@ export const getStoragePath = (): string => { } /** - * Get the storage type - local or s3 + * Get the storage type - local or cloud */ export const getStorageType = (): string => { return process.env.STORAGE_TYPE ? process.env.STORAGE_TYPE : 'local' } -export const removeFilesFromStorage = async (...paths: string[]) => { - const storageType = getStorageType() - if (storageType === 's3') { - let Key = paths.reduce((acc, cur) => acc + '/' + cur, '') - // remove the first '/' if it exists - if (Key.startsWith('/')) { - Key = Key.substring(1) - } - - await _deleteS3Folder(Key) - - // check folder size after deleting all the files - const totalSize = await getS3StorageSize(paths[0]) - return { totalSize: totalSize / 1024 / 1024 } - } else if (storageType === 'gcs') { - const { bucket } = getGcsClient() - const normalizedPath = paths.map((p) => p.replace(/\\/g, '/')).join('/') - await bucket.deleteFiles({ prefix: `${normalizedPath}/` }) - - const totalSize = await getGCSStorageSize(paths[0]) - return { totalSize: totalSize / 1024 / 1024 } - } else { - const directory = path.join(getStoragePath(), ...paths.map(_sanitizeFilename)) - await _deleteLocalFolderRecursive(directory) - - const totalSize = await dirSize(path.join(getStoragePath(), paths[0])) - - return { totalSize: totalSize / 1024 / 1024 } - } +export const removeFilesFromStorage = async (...paths: string[]): Promise => { + const provider = StorageProviderFactory.getProvider() + return provider.removeFilesFromStorage(...paths) } -export const removeSpecificFileFromUpload = async (filePath: string) => { - const storageType = getStorageType() - if (storageType === 's3') { - let Key = filePath - // remove the first '/' if it exists - if (Key.startsWith('/')) { - Key = Key.substring(1) - } - await _deleteS3Folder(Key) - } else if (storageType === 'gcs') { - const { bucket } = getGcsClient() - await bucket.file(filePath).delete() - } else { - fs.unlinkSync(filePath) - } +export const removeSpecificFileFromUpload = async (filePath: string): Promise => { + const provider = StorageProviderFactory.getProvider() + return provider.removeSpecificFileFromUpload(filePath) } -export const removeSpecificFileFromStorage = async (...paths: string[]) => { - const storageType = getStorageType() - if (storageType === 's3') { - let Key = paths.reduce((acc, cur) => acc + '/' + cur, '') - // remove the first '/' if it exists - if (Key.startsWith('/')) { - Key = Key.substring(1) - } - await _deleteS3Folder(Key) - - // check folder size after deleting all the files - const totalSize = await getS3StorageSize(paths[0]) - return { totalSize: totalSize / 1024 / 1024 } - } else if (storageType === 'gcs') { - const { bucket } = getGcsClient() - const fileName = paths.pop() - if (fileName) { - const sanitizedFilename = _sanitizeFilename(fileName) - paths.push(sanitizedFilename) - } - const normalizedPath = paths.map((p) => p.replace(/\\/g, '/')).join('/') - await bucket.file(normalizedPath).delete() - - const totalSize = await getGCSStorageSize(paths[0]) - return { totalSize: totalSize / 1024 / 1024 } - } else { - const fileName = paths.pop() - if (fileName) { - const sanitizedFilename = _sanitizeFilename(fileName) - paths.push(sanitizedFilename) - } - const file = path.join(getStoragePath(), ...paths.map(_sanitizeFilename)) - // check if file exists, if not skip delete - // this might happen when user tries to delete a document loader but the attached file is already deleted - const stat = fs.statSync(file, { throwIfNoEntry: false }) - if (stat && stat.isFile()) { - fs.unlinkSync(file) - } - - const totalSize = await dirSize(path.join(getStoragePath(), paths[0])) - return { totalSize: totalSize / 1024 / 1024 } - } +export const removeSpecificFileFromStorage = async (...paths: string[]): Promise => { + const provider = StorageProviderFactory.getProvider() + return provider.removeSpecificFileFromStorage(...paths) } -export const removeFolderFromStorage = async (...paths: string[]) => { - const storageType = getStorageType() - if (storageType === 's3') { - let Key = paths.reduce((acc, cur) => acc + '/' + cur, '') - // remove the first '/' if it exists - if (Key.startsWith('/')) { - Key = Key.substring(1) - } - await _deleteS3Folder(Key) - - // check folder size after deleting all the files - const totalSize = await getS3StorageSize(paths[0]) - return { totalSize: totalSize / 1024 / 1024 } - } else if (storageType === 'gcs') { - const { bucket } = getGcsClient() - const normalizedPath = paths.map((p) => p.replace(/\\/g, '/')).join('/') - await bucket.deleteFiles({ prefix: `${normalizedPath}/` }) - - const totalSize = await getGCSStorageSize(paths[0]) - return { totalSize: totalSize / 1024 / 1024 } - } else { - const directory = path.join(getStoragePath(), ...paths.map(_sanitizeFilename)) - await _deleteLocalFolderRecursive(directory, true) - - const totalSize = await dirSize(path.join(getStoragePath(), paths[0])) - return { totalSize: totalSize / 1024 / 1024 } - } -} - -const _deleteLocalFolderRecursive = async (directory: string, deleteParentChatflowFolder?: boolean) => { - try { - // Check if the path exists - await fs.promises.access(directory) - - if (deleteParentChatflowFolder) { - await fs.promises.rmdir(directory, { recursive: true }) - } - - // Get stats of the path to determine if it's a file or directory - const stats = await fs.promises.stat(directory) - - if (stats.isDirectory()) { - // Read all directory contents - const files = await fs.promises.readdir(directory) - - // Recursively delete all contents - for (const file of files) { - const currentPath = path.join(directory, file) - await _deleteLocalFolderRecursive(currentPath) // Recursive call - } - - // Delete the directory itself after emptying it - await fs.promises.rmdir(directory, { recursive: true }) - } else { - // If it's a file, delete it directly - await fs.promises.unlink(directory) - } - } catch (error) { - // Error handling - } -} - -const _deleteS3Folder = async (location: string) => { - let count = 0 // number of files deleted - const { s3Client, Bucket } = getS3Config() - - async function recursiveS3Delete(token?: any) { - // get the files - const listCommand = new ListObjectsV2Command({ - Bucket: Bucket, - Prefix: location, - ContinuationToken: token - }) - let list = await s3Client.send(listCommand) - if (list.KeyCount) { - const deleteCommand = new DeleteObjectsCommand({ - Bucket: Bucket, - Delete: { - Objects: list.Contents?.map((item) => ({ Key: item.Key })), - Quiet: false - } - }) - let deleted = await s3Client.send(deleteCommand) - // @ts-ignore - count += deleted.Deleted.length - - if (deleted.Errors) { - deleted.Errors.map((error: any) => console.error(`${error.Key} could not be deleted - ${error.Code}`)) - } - } - // repeat if more files to delete - if (list.NextContinuationToken) { - await recursiveS3Delete(list.NextContinuationToken) - } - // return total deleted count when finished - return `${count} files deleted from S3` - } - - // start the recursive function - return recursiveS3Delete() +export const removeFolderFromStorage = async (...paths: string[]): Promise => { + const provider = StorageProviderFactory.getProvider() + return provider.removeFolderFromStorage(...paths) } export const streamStorageFile = async ( @@ -747,375 +90,14 @@ export const streamStorageFile = async ( fileName: string, orgId: string ): Promise => { - // Validate chatflowId - if (!chatflowId || !isValidUUID(chatflowId)) { - throw new Error('Invalid chatflowId format - must be a valid UUID') - } - - if (!chatId) { - throw new Error('chatId is missing') - } - - // Check for path traversal attempts - if (isPathTraversal(chatflowId) || isPathTraversal(chatId)) { - throw new Error('Invalid path characters detected in chatflowId or chatId') - } - - if (!fileName || isUnsafeFilePath(fileName)) { - throw new Error('Invalid or unsafe fileName detected') - } - - const storageType = getStorageType() - const sanitizedFilename = sanitize(fileName) - - if (!sanitizedFilename || sanitizedFilename.includes('/') || sanitizedFilename.includes('\\')) { - throw new Error('Invalid filename after sanitization') - } - - if (storageType === 's3') { - const { s3Client, Bucket } = getS3Config() - - const Key = orgId + '/' + chatflowId + '/' + chatId + '/' + sanitizedFilename - const getParams = { - Bucket, - Key - } - try { - const response = await s3Client.send(new GetObjectCommand(getParams)) - const body = response.Body - if (body instanceof Readable) { - const blob = await body.transformToByteArray() - return Buffer.from(blob) - } - } catch (error) { - // Fallback: Check if file exists without orgId - const fallbackKey = chatflowId + '/' + chatId + '/' + sanitizedFilename - try { - const fallbackParams = { - Bucket, - Key: fallbackKey - } - const fallbackResponse = await s3Client.send(new GetObjectCommand(fallbackParams)) - const fallbackBody = fallbackResponse.Body - - // If found, copy to correct location with orgId - if (fallbackBody) { - // Get the file content - let fileContent: Buffer - if (fallbackBody instanceof Readable) { - const blob = await fallbackBody.transformToByteArray() - fileContent = Buffer.from(blob) - } else { - // @ts-ignore - fileContent = Buffer.concat(fallbackBody.toArray()) - } - - // Move to correct location with orgId - const putObjCmd = new PutObjectCommand({ - Bucket, - Key, - Body: fileContent - }) - await s3Client.send(putObjCmd) - - // Delete the old file - await s3Client.send( - new DeleteObjectsCommand({ - Bucket, - Delete: { - Objects: [{ Key: fallbackKey }], - Quiet: false - } - }) - ) - - // Check if the directory is empty and delete recursively if needed - await _cleanEmptyS3Folders(s3Client, Bucket, chatflowId) - - return fileContent - } - } catch (fallbackError) { - // File not found in fallback location either - throw new Error(`File ${fileName} not found`) - } - } - } else if (storageType === 'gcs') { - const { bucket } = getGcsClient() - const normalizedChatflowId = chatflowId.replace(/\\/g, '/') - const normalizedChatId = chatId.replace(/\\/g, '/') - const normalizedFilename = sanitizedFilename.replace(/\\/g, '/') - const filePath = `${orgId}/${normalizedChatflowId}/${normalizedChatId}/${normalizedFilename}` - - try { - const [buffer] = await bucket.file(filePath).download() - return buffer - } catch (error) { - // Fallback: Check if file exists without orgId - const fallbackPath = `${normalizedChatflowId}/${normalizedChatId}/${normalizedFilename}` - try { - const fallbackFile = bucket.file(fallbackPath) - const [buffer] = await fallbackFile.download() - - // If found, copy to correct location with orgId - if (buffer) { - const file = bucket.file(filePath) - await new Promise((resolve, reject) => { - file.createWriteStream() - .on('error', (err) => reject(err)) - .on('finish', () => resolve()) - .end(buffer) - }) - - // Delete the old file - await fallbackFile.delete() - - // Check if the directory is empty and delete recursively if needed - await _cleanEmptyGCSFolders(bucket, normalizedChatflowId) - - return buffer - } - } catch (fallbackError) { - // File not found in fallback location either - throw new Error(`File ${fileName} not found`) - } - } - } else { - const filePath = path.join(getStoragePath(), orgId, chatflowId, chatId, sanitizedFilename) - //raise error if file path is not absolute - if (!path.isAbsolute(filePath)) throw new Error(`Invalid file path`) - //raise error if file path contains '..' - if (filePath.includes('..')) throw new Error(`Invalid file path`) - //only return from the storage folder - if (!filePath.startsWith(getStoragePath())) throw new Error(`Invalid file path`) - - if (fs.existsSync(filePath)) { - return fs.createReadStream(filePath) - } else { - // Fallback: Check if file exists without orgId - const fallbackPath = path.join(getStoragePath(), chatflowId, chatId, sanitizedFilename) - - if (fs.existsSync(fallbackPath)) { - // Create directory if it doesn't exist - const dir = path.dirname(filePath) - if (!fs.existsSync(dir)) { - fs.mkdirSync(dir, { recursive: true }) - } - - // Copy file to correct location with orgId - fs.copyFileSync(fallbackPath, filePath) - - // Delete the old file - fs.unlinkSync(fallbackPath) - - // Clean up empty directories recursively - _cleanEmptyLocalFolders(path.join(getStoragePath(), chatflowId, chatId)) - - return fs.createReadStream(filePath) - } else { - throw new Error(`File ${fileName} not found`) - } - } - } + const provider = StorageProviderFactory.getProvider() + return provider.streamStorageFile(chatflowId, chatId, fileName, orgId) } /** - * Check if a local directory is empty and delete it if so, - * then check parent directories recursively + * Get the total storage size for an organization (unified across all providers) */ -const _cleanEmptyLocalFolders = (dirPath: string) => { - try { - // Stop if we reach the storage root - if (dirPath === getStoragePath()) return - - // Check if directory exists - if (!fs.existsSync(dirPath)) return - - // Read directory contents - const files = fs.readdirSync(dirPath) - - // If directory is empty, delete it and check parent - if (files.length === 0) { - fs.rmdirSync(dirPath) - // Recursively check parent directory - _cleanEmptyLocalFolders(path.dirname(dirPath)) - } - } catch (error) { - // Ignore errors during cleanup - console.error('Error cleaning empty folders:', error) - } -} - -/** - * Check if an S3 "folder" is empty and delete it recursively - */ -const _cleanEmptyS3Folders = async (s3Client: S3Client, Bucket: string, prefix: string) => { - try { - // Skip if prefix is empty - if (!prefix) return - - // List objects in this "folder" - const listCmd = new ListObjectsV2Command({ - Bucket, - Prefix: prefix + '/', - Delimiter: '/' - }) - - const response = await s3Client.send(listCmd) - - // If folder is empty (only contains common prefixes but no files) - if ( - (response.Contents?.length === 0 || !response.Contents) && - (response.CommonPrefixes?.length === 0 || !response.CommonPrefixes) - ) { - // Delete the folder marker if it exists - await s3Client.send( - new DeleteObjectsCommand({ - Bucket, - Delete: { - Objects: [{ Key: prefix + '/' }], - Quiet: true - } - }) - ) - - // Recursively check parent folder - const parentPrefix = prefix.substring(0, prefix.lastIndexOf('/')) - if (parentPrefix) { - await _cleanEmptyS3Folders(s3Client, Bucket, parentPrefix) - } - } - } catch (error) { - // Ignore errors during cleanup - console.error('Error cleaning empty S3 folders:', error) - } -} - -/** - * Check if a GCS "folder" is empty and delete recursively if so - */ -const _cleanEmptyGCSFolders = async (bucket: any, prefix: string) => { - try { - // Skip if prefix is empty - if (!prefix) return - - // List files with this prefix - const [files] = await bucket.getFiles({ - prefix: prefix + '/', - delimiter: '/' - }) - - // If folder is empty (no files) - if (files.length === 0) { - // Delete the folder marker if it exists - try { - await bucket.file(prefix + '/').delete() - } catch (err) { - // Folder marker might not exist, ignore - } - - // Recursively check parent folder - const parentPrefix = prefix.substring(0, prefix.lastIndexOf('/')) - if (parentPrefix) { - await _cleanEmptyGCSFolders(bucket, parentPrefix) - } - } - } catch (error) { - // Ignore errors during cleanup - console.error('Error cleaning empty GCS folders:', error) - } -} - -export const getGCSStorageSize = async (orgId: string): Promise => { - const { bucket } = getGcsClient() - let totalSize = 0 - - const [files] = await bucket.getFiles({ prefix: orgId }) - - for (const file of files) { - const size = file.metadata.size - // Handle different types that size could be - if (typeof size === 'string') { - totalSize += parseInt(size, 10) || 0 - } else if (typeof size === 'number') { - totalSize += size - } - } - - return totalSize -} - -export const getGcsClient = () => { - const pathToGcsCredential = process.env.GOOGLE_CLOUD_STORAGE_CREDENTIAL - const projectId = process.env.GOOGLE_CLOUD_STORAGE_PROJ_ID - const bucketName = process.env.GOOGLE_CLOUD_STORAGE_BUCKET_NAME - - if (!bucketName) { - throw new Error('GOOGLE_CLOUD_STORAGE_BUCKET_NAME env variable is required') - } - - const storageConfig = { - ...(pathToGcsCredential ? { keyFilename: pathToGcsCredential } : {}), - ...(projectId ? { projectId } : {}) - } - - const storage = new Storage(storageConfig) - const bucket = storage.bucket(bucketName) - return { storage, bucket } -} - -export const getS3StorageSize = async (orgId: string): Promise => { - const { s3Client, Bucket } = getS3Config() - const getCmd = new ListObjectsCommand({ - Bucket, - Prefix: orgId - }) - const headObj = await s3Client.send(getCmd) - let totalSize = 0 - for (const obj of headObj.Contents || []) { - totalSize += obj.Size || 0 - } - return totalSize -} - -export const getS3Config = () => { - const accessKeyId = process.env.S3_STORAGE_ACCESS_KEY_ID - const secretAccessKey = process.env.S3_STORAGE_SECRET_ACCESS_KEY - const region = process.env.S3_STORAGE_REGION - const Bucket = process.env.S3_STORAGE_BUCKET_NAME - const customURL = process.env.S3_ENDPOINT_URL - const forcePathStyle = process.env.S3_FORCE_PATH_STYLE === 'true' ? true : false - - if (!region || region.trim() === '' || !Bucket || Bucket.trim() === '') { - throw new Error('S3 storage configuration is missing') - } - - const s3Config: S3ClientConfig = { - region: region, - forcePathStyle: forcePathStyle - } - - // Only include endpoint if customURL is not empty - if (customURL && customURL.trim() !== '') { - s3Config.endpoint = customURL - } - - if (accessKeyId && accessKeyId.trim() !== '' && secretAccessKey && secretAccessKey.trim() !== '') { - s3Config.credentials = { - accessKeyId: accessKeyId, - secretAccessKey: secretAccessKey - } - } - - const s3Client = new S3Client(s3Config) - - return { s3Client, Bucket } -} - -const _sanitizeFilename = (filename: string): string => { - if (filename) { - let sanitizedFilename = sanitize(filename) - // remove all leading . - return sanitizedFilename.replace(/^\.+/, '') - } - return '' +export const getStorageSize = async (orgId: string): Promise => { + const provider = StorageProviderFactory.getProvider() + return provider.getStorageSize(orgId) } diff --git a/packages/server/.env.example b/packages/server/.env.example index 46666cf8b..01f0af7eb 100644 --- a/packages/server/.env.example +++ b/packages/server/.env.example @@ -49,7 +49,7 @@ PORT=3000 ############################################## STORAGE ##################################################### ############################################################################################################ -# STORAGE_TYPE=local (local | s3 | gcs) +# STORAGE_TYPE=local (local | s3 | gcs | azure) # BLOB_STORAGE_PATH=/your_storage_path/.flowise/storage # S3_STORAGE_BUCKET_NAME=flowise # S3_STORAGE_ACCESS_KEY_ID= @@ -61,7 +61,12 @@ PORT=3000 # GOOGLE_CLOUD_STORAGE_PROJ_ID= # GOOGLE_CLOUD_STORAGE_BUCKET_NAME= # GOOGLE_CLOUD_UNIFORM_BUCKET_ACCESS=true - +# Azure Blob Storage (when STORAGE_TYPE=azure) +# Provide EITHER a connection string OR account name + key +# AZURE_BLOB_STORAGE_CONNECTION_STRING= +# AZURE_BLOB_STORAGE_ACCOUNT_NAME= +# AZURE_BLOB_STORAGE_ACCOUNT_KEY= +# AZURE_BLOB_STORAGE_CONTAINER_NAME= ############################################################################################################ ############################################## SETTINGS #################################################### diff --git a/packages/server/package.json b/packages/server/package.json index 9171e0483..e66afc208 100644 --- a/packages/server/package.json +++ b/packages/server/package.json @@ -120,6 +120,7 @@ "moment": "^2.29.3", "moment-timezone": "^0.5.34", "multer": "^2.0.2", + "multer-azure-blob-storage": "^1.2.0", "multer-cloud-storage": "^4.0.0", "multer-s3": "^3.0.1", "mysql2": "^3.11.3", @@ -147,6 +148,7 @@ "typeorm": "^0.3.6", "uuid": "^9.0.1", "winston": "^3.9.0", + "winston-azure-blob": "^1.5.0", "winston-daily-rotate-file": "^5.0.0" }, "devDependencies": { diff --git a/packages/server/src/commands/base.ts b/packages/server/src/commands/base.ts index e09a1f0dc..3fba54422 100644 --- a/packages/server/src/commands/base.ts +++ b/packages/server/src/commands/base.ts @@ -68,6 +68,10 @@ export abstract class BaseCommand extends Command { GOOGLE_CLOUD_STORAGE_PROJ_ID: Flags.string(), GOOGLE_CLOUD_STORAGE_BUCKET_NAME: Flags.string(), GOOGLE_CLOUD_UNIFORM_BUCKET_ACCESS: Flags.string(), + AZURE_BLOB_STORAGE_CONNECTION_STRING: Flags.string(), + AZURE_BLOB_STORAGE_ACCOUNT_NAME: Flags.string(), + AZURE_BLOB_STORAGE_ACCOUNT_KEY: Flags.string(), + AZURE_BLOB_STORAGE_CONTAINER_NAME: Flags.string(), // Credentials / Secret Keys SECRETKEY_STORAGE_TYPE: Flags.string(), diff --git a/packages/server/src/utils/index.ts b/packages/server/src/utils/index.ts index f74917d0c..7ea3adbfd 100644 --- a/packages/server/src/utils/index.ts +++ b/packages/server/src/utils/index.ts @@ -38,13 +38,11 @@ import { IMessage, FlowiseMemory, IFileUpload, - getS3Config + StorageProviderFactory } from 'flowise-components' import { randomBytes } from 'crypto' import { AES, enc } from 'crypto-js' -import multer from 'multer' -import multerS3 from 'multer-s3' -import MulterGoogleCloudStorage from 'multer-cloud-storage' + import { ChatFlow } from '../database/entities/ChatFlow' import { ChatMessage } from '../database/entities/ChatMessage' import { Credential } from '../database/entities/Credential' @@ -1997,38 +1995,8 @@ export function generateId() { } export const getMulterStorage = () => { - const storageType = process.env.STORAGE_TYPE ? process.env.STORAGE_TYPE : 'local' - - if (storageType === 's3') { - const s3Client = getS3Config().s3Client - const Bucket = getS3Config().Bucket - - const upload = multer({ - storage: multerS3({ - s3: s3Client, - bucket: Bucket, - metadata: function (req, file, cb) { - cb(null, { fieldName: file.fieldname, originalName: file.originalname }) - }, - key: function (req, file, cb) { - cb(null, `${generateId()}`) - } - }) - }) - return upload - } else if (storageType === 'gcs') { - return multer({ - storage: new MulterGoogleCloudStorage({ - projectId: process.env.GOOGLE_CLOUD_STORAGE_PROJ_ID, - bucket: process.env.GOOGLE_CLOUD_STORAGE_BUCKET_NAME, - keyFilename: process.env.GOOGLE_CLOUD_STORAGE_CREDENTIAL, - uniformBucketLevelAccess: Boolean(process.env.GOOGLE_CLOUD_UNIFORM_BUCKET_ACCESS) ?? true, - destination: `uploads/${generateId()}` - }) - }) - } else { - return multer({ dest: getUploadPath() }) - } + const provider = StorageProviderFactory.getProvider() + return provider.getMulterStorage() } /** diff --git a/packages/server/src/utils/logger.ts b/packages/server/src/utils/logger.ts index 71e8e309c..5db5bc93a 100644 --- a/packages/server/src/utils/logger.ts +++ b/packages/server/src/utils/logger.ts @@ -1,100 +1,18 @@ -import * as path from 'path' import * as fs from 'fs' import config from './config' // should be replaced by node-config or similar import { createLogger, transports, format } from 'winston' import { NextFunction, Request, Response } from 'express' -import DailyRotateFile from 'winston-daily-rotate-file' -import { S3ClientConfig } from '@aws-sdk/client-s3' -import { LoggingWinston } from '@google-cloud/logging-winston' - -const { S3StreamLogger } = require('s3-streamlogger') +import { StorageProviderFactory } from 'flowise-components' const { combine, timestamp, printf, errors } = format -let s3ServerStream: any -let s3ErrorStream: any -let s3ServerReqStream: any - -let gcsServerStream: any -let gcsErrorStream: any -let gcsServerReqStream: any - let requestLogger: any -if (process.env.STORAGE_TYPE === 's3') { - const accessKeyId = process.env.S3_STORAGE_ACCESS_KEY_ID - const secretAccessKey = process.env.S3_STORAGE_SECRET_ACCESS_KEY - const region = process.env.S3_STORAGE_REGION - const s3Bucket = process.env.S3_STORAGE_BUCKET_NAME - const customURL = process.env.S3_ENDPOINT_URL - const forcePathStyle = process.env.S3_FORCE_PATH_STYLE === 'true' +const provider = StorageProviderFactory.getProvider() +const serverTransports = provider.getLoggerTransports('server', config) +const errorTransports = provider.getLoggerTransports('error', config) +const requestTransports = provider.getLoggerTransports('requests', config) - if (!region || region.trim() === '' || !s3Bucket || s3Bucket.trim() === '') { - throw new Error('S3 storage configuration is missing') - } - - const s3Config: S3ClientConfig = { - region: region, - forcePathStyle: forcePathStyle - } - - // Only include endpoint if customURL is not empty - if (customURL && customURL.trim() !== '') { - s3Config.endpoint = customURL - } - - if (accessKeyId && accessKeyId.trim() !== '' && secretAccessKey && secretAccessKey.trim() !== '') { - s3Config.credentials = { - accessKeyId: accessKeyId, - secretAccessKey: secretAccessKey - } - } - - s3ServerStream = new S3StreamLogger({ - bucket: s3Bucket, - folder: 'logs/server', - name_format: `server-%Y-%m-%d-%H-%M-%S-%L.log`, - config: s3Config - }) - - s3ErrorStream = new S3StreamLogger({ - bucket: s3Bucket, - folder: 'logs/error', - name_format: `server-error-%Y-%m-%d-%H-%M-%S-%L.log`, - config: s3Config - }) - - s3ServerReqStream = new S3StreamLogger({ - bucket: s3Bucket, - folder: 'logs/requests', - name_format: `server-requests-%Y-%m-%d-%H-%M-%S-%L.log.jsonl`, - config: s3Config - }) -} - -if (process.env.STORAGE_TYPE === 'gcs') { - const config = { - projectId: process.env.GOOGLE_CLOUD_STORAGE_PROJ_ID, - keyFilename: process.env.GOOGLE_CLOUD_STORAGE_CREDENTIAL, - defaultCallback: (err: any) => { - if (err) { - console.error('Error logging to GCS: ' + err) - } - } - } - gcsServerStream = new LoggingWinston({ - ...config, - logName: 'server' - }) - gcsErrorStream = new LoggingWinston({ - ...config, - logName: 'error' - }) - gcsServerReqStream = new LoggingWinston({ - ...config, - logName: 'requests' - }) -} // expect the log dir be relative to the projects root const logDir = config.logging.dir @@ -117,52 +35,13 @@ const logger = createLogger({ package: 'server' }, exitOnError: false, - transports: [ - new transports.Console(), - ...(!process.env.STORAGE_TYPE || process.env.STORAGE_TYPE === 'local' - ? [ - new DailyRotateFile({ - filename: path.join(logDir, config.logging.server.filename ?? 'server-%DATE%.log'), - datePattern: 'YYYY-MM-DD-HH', - maxSize: '20m', - level: config.logging.server.level ?? 'info' - }) - ] - : []), - ...(process.env.STORAGE_TYPE === 's3' - ? [ - new transports.Stream({ - stream: s3ServerStream - }) - ] - : []), - ...(process.env.STORAGE_TYPE === 'gcs' ? [gcsServerStream] : []) - ], - exceptionHandlers: [ - ...(process.env.DEBUG && process.env.DEBUG === 'true' ? [new transports.Console()] : []), - ...(process.env.STORAGE_TYPE === 's3' - ? [ - new transports.Stream({ - stream: s3ErrorStream - }) - ] - : []), - ...(process.env.STORAGE_TYPE === 'gcs' ? [gcsErrorStream] : []) - ], + transports: [new transports.Console(), ...serverTransports], + exceptionHandlers: [...(process.env.DEBUG && process.env.DEBUG === 'true' ? [new transports.Console()] : []), ...errorTransports], rejectionHandlers: [ ...(process.env.DEBUG && process.env.DEBUG === 'true' ? [new transports.Console()] : []), - ...(process.env.STORAGE_TYPE === 's3' - ? [ - new transports.Stream({ - stream: s3ErrorStream - }) - ] - : []), - ...(process.env.STORAGE_TYPE === 'gcs' ? [gcsErrorStream] : []), + ...errorTransports, // Always provide a fallback rejection handler when no other handlers are configured - ...((!process.env.DEBUG || process.env.DEBUG !== 'true') && process.env.STORAGE_TYPE !== 's3' && process.env.STORAGE_TYPE !== 'gcs' - ? [new transports.Console()] - : []) + ...((!process.env.DEBUG || process.env.DEBUG !== 'true') && errorTransports.length === 0 ? [new transports.Console()] : []) ] }) @@ -171,25 +50,7 @@ requestLogger = createLogger({ defaultMeta: { package: 'server' }, - transports: [ - ...(process.env.DEBUG && process.env.DEBUG === 'true' ? [new transports.Console()] : []), - ...(!process.env.STORAGE_TYPE || process.env.STORAGE_TYPE === 'local' - ? [ - new transports.File({ - filename: path.join(logDir, config.logging.express.filename ?? 'server-requests.log.jsonl'), - level: config.logging.express.level ?? 'debug' - }) - ] - : []), - ...(process.env.STORAGE_TYPE === 's3' - ? [ - new transports.Stream({ - stream: s3ServerReqStream - }) - ] - : []), - ...(process.env.STORAGE_TYPE === 'gcs' ? [gcsServerReqStream] : []) - ] + transports: [...(process.env.DEBUG && process.env.DEBUG === 'true' ? [new transports.Console()] : []), ...requestTransports] }) function getSensitiveBodyFields(): string[] { diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index f1257120a..8257b779c 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -251,6 +251,9 @@ importers: '@aws-sdk/client-sts': specifier: ^3.699.0 version: 3.726.1 + '@azure/storage-blob': + specifier: ^12.29.0 + version: 12.31.0 '@datastax/astra-db-ts': specifier: 1.5.0 version: 1.5.0 @@ -313,7 +316,7 @@ importers: version: 0.0.7(encoding@0.1.13)(openai@4.96.0(encoding@0.1.13)(ws@8.18.3(bufferutil@4.0.8)(utf-8-validate@6.0.4))(zod@3.22.4)) '@langchain/community': specifier: ^0.3.47 - version: 0.3.49(d32382e756a310688ffd878bb799886f) + version: 0.3.49(96a2dea491375c22412401f1555fc2c2) '@langchain/core': specifier: 0.3.61 version: 0.3.61(openai@4.96.0(encoding@0.1.13)(ws@8.18.3(bufferutil@4.0.8)(utf-8-validate@6.0.4))(zod@3.22.4)) @@ -358,7 +361,7 @@ importers: version: 0.0.1(@langchain/core@0.3.61(openai@4.96.0(encoding@0.1.13)(ws@8.18.3(bufferutil@4.0.8)(utf-8-validate@6.0.4))(zod@3.22.4)))(encoding@0.1.13)(ws@8.18.3(bufferutil@4.0.8)(utf-8-validate@6.0.4)) '@mem0/community': specifier: ^0.0.1 - version: 0.0.1(5cc11fc2ac52307d4013e43691e7b64a) + version: 0.0.1(5a79e038a0f9851c2336c9f7472dc8f5) '@mendable/firecrawl-js': specifier: ^1.18.2 version: 1.25.1 @@ -900,6 +903,9 @@ importers: multer: specifier: ^2.0.2 version: 2.0.2 + multer-azure-blob-storage: + specifier: ^1.2.0 + version: 1.2.0 multer-cloud-storage: specifier: ^4.0.0 version: 4.0.0(encoding@0.1.13) @@ -981,6 +987,9 @@ importers: winston: specifier: ^3.9.0 version: 3.12.0 + winston-azure-blob: + specifier: ^1.5.0 + version: 1.5.0(async@3.2.5)(winston@3.12.0) winston-daily-rotate-file: specifier: ^5.0.0 version: 5.0.0(winston@3.12.0) @@ -2152,6 +2161,61 @@ packages: resolution: {integrity: sha512-6Ed0kmC1NMbuFTEgNmamAUU1h5gShgxL1hBVLbEzUa3trX5aJBz1vU4bXaBTvOYUAnOHtiy1Ml4AMStd6hJnFA==} engines: {node: '>=18.0.0'} + '@azure/abort-controller@2.1.2': + resolution: {integrity: sha512-nBrLsEWm4J2u5LpAPjxADTlq3trDgVZZXHNKabeXZtpq3d3AbN/KGO82R87rdDz5/lYB024rtEf10/q0urNgsA==} + engines: {node: '>=18.0.0'} + + '@azure/core-auth@1.10.1': + resolution: {integrity: sha512-ykRMW8PjVAn+RS6ww5cmK9U2CyH9p4Q88YJwvUslfuMmN98w/2rdGRLPqJYObapBCdzBVeDgYWdJnFPFb7qzpg==} + engines: {node: '>=20.0.0'} + + '@azure/core-client@1.10.1': + resolution: {integrity: sha512-Nh5PhEOeY6PrnxNPsEHRr9eimxLwgLlpmguQaHKBinFYA/RU9+kOYVOQqOrTsCL+KSxrLLl1gD8Dk5BFW/7l/w==} + engines: {node: '>=20.0.0'} + + '@azure/core-http-compat@2.3.2': + resolution: {integrity: sha512-Tf6ltdKzOJEgxZeWLCjMxrxbodB/ZeCbzzA1A2qHbhzAjzjHoBVSUeSl/baT/oHAxhc4qdqVaDKnc2+iE932gw==} + engines: {node: '>=20.0.0'} + peerDependencies: + '@azure/core-client': ^1.10.0 + '@azure/core-rest-pipeline': ^1.22.0 + + '@azure/core-lro@2.7.2': + resolution: {integrity: sha512-0YIpccoX8m/k00O7mDDMdJpbr6mf1yWo2dfmxt5A8XVZVVMz2SSKaEbMCeJRvgQ0IaSlqhjT47p4hVIRRy90xw==} + engines: {node: '>=18.0.0'} + + '@azure/core-paging@1.6.2': + resolution: {integrity: sha512-YKWi9YuCU04B55h25cnOYZHxXYtEvQEbKST5vqRga7hWY9ydd3FZHdeQF8pyh+acWZvppw13M/LMGx0LABUVMA==} + engines: {node: '>=18.0.0'} + + '@azure/core-rest-pipeline@1.22.2': + resolution: {integrity: sha512-MzHym+wOi8CLUlKCQu12de0nwcq9k9Kuv43j4Wa++CsCpJwps2eeBQwD2Bu8snkxTtDKDx4GwjuR9E8yC8LNrg==} + engines: {node: '>=20.0.0'} + + '@azure/core-tracing@1.3.1': + resolution: {integrity: sha512-9MWKevR7Hz8kNzzPLfX4EAtGM2b8mr50HPDBvio96bURP/9C+HjdH3sBlLSNNrvRAr5/k/svoH457gB5IKpmwQ==} + engines: {node: '>=20.0.0'} + + '@azure/core-util@1.13.1': + resolution: {integrity: sha512-XPArKLzsvl0Hf0CaGyKHUyVgF7oDnhKoP85Xv6M4StF/1AhfORhZudHtOyf2s+FcbuQ9dPRAjB8J2KvRRMUK2A==} + engines: {node: '>=20.0.0'} + + '@azure/core-xml@1.5.0': + resolution: {integrity: sha512-D/sdlJBMJfx7gqoj66PKVmhDDaU6TKA49ptcolxdas29X7AfvLTmfAGLjAcIMBK7UZ2o4lygHIqVckOlQU3xWw==} + engines: {node: '>=20.0.0'} + + '@azure/logger@1.3.0': + resolution: {integrity: sha512-fCqPIfOcLE+CGqGPd66c8bZpwAji98tZ4JI9i/mlTNTlsIWslCfpg48s/ypyLxZTump5sypjrKn2/kY7q8oAbA==} + engines: {node: '>=20.0.0'} + + '@azure/storage-blob@12.31.0': + resolution: {integrity: sha512-DBgNv10aCSxopt92DkTDD0o9xScXeBqPKGmR50FPZQaEcH4JLQ+GEOGEDv19V5BMkB7kxr+m4h6il/cCDPvmHg==} + engines: {node: '>=20.0.0'} + + '@azure/storage-common@12.3.0': + resolution: {integrity: sha512-/OFHhy86aG5Pe8dP5tsp+BuJ25JOAl9yaMU3WZbkeoiFMHFtJ7tu5ili7qEdBXNW9G5lDB19trwyI6V49F/8iQ==} + engines: {node: '>=20.0.0'} + '@babel/code-frame@7.26.2': resolution: {integrity: sha512-RJlIHRueQgwWitWgF8OdFYGZX328Ax5BCemNGlqHfplnRT9ESi8JkFlvaVYbS+UubVY6dpv87Fs2u5M29iNFVQ==} engines: {node: '>=6.9.0'} @@ -3556,9 +3620,6 @@ packages: '@emnapi/core@1.8.1': resolution: {integrity: sha512-AvT9QFpxK0Zd8J0jopedNm+w/2fIzvtPKPjqyw9jwvBaReTTqPBk9Hixaz7KbjimP+QNz605/XnjFcDAL2pqBg==} - '@emnapi/runtime@1.3.1': - resolution: {integrity: sha512-kEBmG8KyqtxJZv+ygbEim+KCGtIq1fC22Ms3S4ziXmYKm8uyoLX0MHONVKwp+9opg390VaKRNt4a7A9NwmpNhw==} - '@emnapi/runtime@1.8.1': resolution: {integrity: sha512-mehfKSMWjjNol8659Z8KxEMrdSJDDot5SXMq00dM8BN4o+CLNXQ0xH2V7EchNHV4RmbZLmmPdEaXZc5H2FXmDg==} @@ -4145,67 +4206,79 @@ packages: resolution: {integrity: sha512-9B+taZ8DlyyqzZQnoeIvDVR/2F4EbMepXMc/NdVbkzsJbzkUjhXv/70GQJ7tdLA4YJgNP25zukcxpX2/SueNrA==} cpu: [arm64] os: [linux] + libc: [glibc] '@img/sharp-libvips-linux-arm@1.0.5': resolution: {integrity: sha512-gvcC4ACAOPRNATg/ov8/MnbxFDJqf/pDePbBnuBDcjsI8PssmjoKMAz4LtLaVi+OnSb5FK/yIOamqDwGmXW32g==} cpu: [arm] os: [linux] + libc: [glibc] '@img/sharp-libvips-linux-s390x@1.0.4': resolution: {integrity: sha512-u7Wz6ntiSSgGSGcjZ55im6uvTrOxSIS8/dgoVMoiGE9I6JAfU50yH5BoDlYA1tcuGS7g/QNtetJnxA6QEsCVTA==} cpu: [s390x] os: [linux] + libc: [glibc] '@img/sharp-libvips-linux-x64@1.0.4': resolution: {integrity: sha512-MmWmQ3iPFZr0Iev+BAgVMb3ZyC4KeFc3jFxnNbEPas60e1cIfevbtuyf9nDGIzOaW9PdnDciJm+wFFaTlj5xYw==} cpu: [x64] os: [linux] + libc: [glibc] '@img/sharp-libvips-linuxmusl-arm64@1.0.4': resolution: {integrity: sha512-9Ti+BbTYDcsbp4wfYib8Ctm1ilkugkA/uscUn6UXK1ldpC1JjiXbLfFZtRlBhjPZ5o1NCLiDbg8fhUPKStHoTA==} cpu: [arm64] os: [linux] + libc: [musl] '@img/sharp-libvips-linuxmusl-x64@1.0.4': resolution: {integrity: sha512-viYN1KX9m+/hGkJtvYYp+CCLgnJXwiQB39damAO7WMdKWlIhmYTfHjwSbQeUK/20vY154mwezd9HflVFM1wVSw==} cpu: [x64] os: [linux] + libc: [musl] '@img/sharp-linux-arm64@0.33.5': resolution: {integrity: sha512-JMVv+AMRyGOHtO1RFBiJy/MBsgz0x4AWrT6QoEVVTyh1E39TrCUpTRI7mx9VksGX4awWASxqCYLCV4wBZHAYxA==} engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} cpu: [arm64] os: [linux] + libc: [glibc] '@img/sharp-linux-arm@0.33.5': resolution: {integrity: sha512-JTS1eldqZbJxjvKaAkxhZmBqPRGmxgu+qFKSInv8moZ2AmT5Yib3EQ1c6gp493HvrvV8QgdOXdyaIBrhvFhBMQ==} engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} cpu: [arm] os: [linux] + libc: [glibc] '@img/sharp-linux-s390x@0.33.5': resolution: {integrity: sha512-y/5PCd+mP4CA/sPDKl2961b+C9d+vPAveS33s6Z3zfASk2j5upL6fXVPZi7ztePZ5CuH+1kW8JtvxgbuXHRa4Q==} engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} cpu: [s390x] os: [linux] + libc: [glibc] '@img/sharp-linux-x64@0.33.5': resolution: {integrity: sha512-opC+Ok5pRNAzuvq1AG0ar+1owsu842/Ab+4qvU879ippJBHvyY5n2mxF1izXqkPYlGuP/M556uh53jRLJmzTWA==} engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} cpu: [x64] os: [linux] + libc: [glibc] '@img/sharp-linuxmusl-arm64@0.33.5': resolution: {integrity: sha512-XrHMZwGQGvJg2V/oRSUfSAfjfPxO+4DkiRh6p2AFjLQztWUuY/o8Mq0eMQVIY7HJ1CDQUJlxGGZRw1a5bqmd1g==} engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} cpu: [arm64] os: [linux] + libc: [musl] '@img/sharp-linuxmusl-x64@0.33.5': resolution: {integrity: sha512-WT+d/cgqKkkKySYmqoZ8y3pxx7lx9vVejxW/W4DOFMYVSkErR+w7mf2u8m/y4+xHe7yY9DAXQMWQhpnMuFfScw==} engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0} cpu: [x64] os: [linux] + libc: [musl] '@img/sharp-wasm32@0.33.5': resolution: {integrity: sha512-ykUW4LVGaMcU9lu9thv85CbRMAwfeadCJHRsg2GmeRa/cJxsVY9Rbd57JcMxBkKHag5U/x7TSBpScF4U8ElVzg==} @@ -5561,30 +5634,35 @@ packages: engines: {node: '>= 10'} cpu: [arm64] os: [linux] + libc: [glibc] '@napi-rs/canvas-linux-arm64-musl@0.1.73': resolution: {integrity: sha512-lX0z2bNmnk1PGZ+0a9OZwI2lPPvWjRYzPqvEitXX7lspyLFrOzh2kcQiLL7bhyODN23QvfriqwYqp5GreSzVvA==} engines: {node: '>= 10'} cpu: [arm64] os: [linux] + libc: [musl] '@napi-rs/canvas-linux-riscv64-gnu@0.1.73': resolution: {integrity: sha512-QDQgMElwxAoADsSR3UYvdTTQk5XOyD9J5kq15Z8XpGwpZOZsSE0zZ/X1JaOtS2x+HEZL6z1S6MF/1uhZFZb5ig==} engines: {node: '>= 10'} cpu: [riscv64] os: [linux] + libc: [glibc] '@napi-rs/canvas-linux-x64-gnu@0.1.73': resolution: {integrity: sha512-wbzLJrTalQrpyrU1YRrO6w6pdr5vcebbJa+Aut5QfTaW9eEmMb1WFG6l1V+cCa5LdHmRr8bsvl0nJDU/IYDsmw==} engines: {node: '>= 10'} cpu: [x64] os: [linux] + libc: [glibc] '@napi-rs/canvas-linux-x64-musl@0.1.73': resolution: {integrity: sha512-xbfhYrUufoTAKvsEx2ZUN4jvACabIF0h1F5Ik1Rk4e/kQq6c+Dwa5QF0bGrfLhceLpzHT0pCMGMDeQKQrcUIyA==} engines: {node: '>= 10'} cpu: [x64] os: [linux] + libc: [musl] '@napi-rs/canvas-win32-x64-msvc@0.1.73': resolution: {integrity: sha512-YQmHXBufFBdWqhx+ympeTPkMfs3RNxaOgWm59vyjpsub7Us07BwCcmu1N5kildhO8Fm0syoI2kHnzGkJBLSvsg==} @@ -6449,56 +6527,67 @@ packages: resolution: {integrity: sha512-hLrmRl53prCcD+YXTfNvXd776HTxNh8wPAMllusQ+amcQmtgo3V5i/nkhPN6FakW+QVLoUUr2AsbtIRPFU3xIA==} cpu: [arm] os: [linux] + libc: [glibc] '@rollup/rollup-linux-arm-musleabihf@4.45.0': resolution: {integrity: sha512-XBKGSYcrkdiRRjl+8XvrUR3AosXU0NvF7VuqMsm7s5nRy+nt58ZMB19Jdp1RdqewLcaYnpk8zeVs/4MlLZEJxw==} cpu: [arm] os: [linux] + libc: [musl] '@rollup/rollup-linux-arm64-gnu@4.45.0': resolution: {integrity: sha512-fRvZZPUiBz7NztBE/2QnCS5AtqLVhXmUOPj9IHlfGEXkapgImf4W9+FSkL8cWqoAjozyUzqFmSc4zh2ooaeF6g==} cpu: [arm64] os: [linux] + libc: [glibc] '@rollup/rollup-linux-arm64-musl@4.45.0': resolution: {integrity: sha512-Btv2WRZOcUGi8XU80XwIvzTg4U6+l6D0V6sZTrZx214nrwxw5nAi8hysaXj/mctyClWgesyuxbeLylCBNauimg==} cpu: [arm64] os: [linux] + libc: [musl] '@rollup/rollup-linux-loongarch64-gnu@4.45.0': resolution: {integrity: sha512-Li0emNnwtUZdLwHjQPBxn4VWztcrw/h7mgLyHiEI5Z0MhpeFGlzaiBHpSNVOMB/xucjXTTcO+dhv469Djr16KA==} cpu: [loong64] os: [linux] + libc: [glibc] '@rollup/rollup-linux-powerpc64le-gnu@4.45.0': resolution: {integrity: sha512-sB8+pfkYx2kvpDCfd63d5ScYT0Fz1LO6jIb2zLZvmK9ob2D8DeVqrmBDE0iDK8KlBVmsTNzrjr3G1xV4eUZhSw==} cpu: [ppc64] os: [linux] + libc: [glibc] '@rollup/rollup-linux-riscv64-gnu@4.45.0': resolution: {integrity: sha512-5GQ6PFhh7E6jQm70p1aW05G2cap5zMOvO0se5JMecHeAdj5ZhWEHbJ4hiKpfi1nnnEdTauDXxPgXae/mqjow9w==} cpu: [riscv64] os: [linux] + libc: [glibc] '@rollup/rollup-linux-riscv64-musl@4.45.0': resolution: {integrity: sha512-N/euLsBd1rekWcuduakTo/dJw6U6sBP3eUq+RXM9RNfPuWTvG2w/WObDkIvJ2KChy6oxZmOSC08Ak2OJA0UiAA==} cpu: [riscv64] os: [linux] + libc: [musl] '@rollup/rollup-linux-s390x-gnu@4.45.0': resolution: {integrity: sha512-2l9sA7d7QdikL0xQwNMO3xURBUNEWyHVHfAsHsUdq+E/pgLTUcCE+gih5PCdmyHmfTDeXUWVhqL0WZzg0nua3g==} cpu: [s390x] os: [linux] + libc: [glibc] '@rollup/rollup-linux-x64-gnu@4.45.0': resolution: {integrity: sha512-XZdD3fEEQcwG2KrJDdEQu7NrHonPxxaV0/w2HpvINBdcqebz1aL+0vM2WFJq4DeiAVT6F5SUQas65HY5JDqoPw==} cpu: [x64] os: [linux] + libc: [glibc] '@rollup/rollup-linux-x64-musl@4.45.0': resolution: {integrity: sha512-7ayfgvtmmWgKWBkCGg5+xTQ0r5V1owVm67zTrsEY1008L5ro7mCyGYORomARt/OquB9KY7LpxVBZes+oSniAAQ==} cpu: [x64] os: [linux] + libc: [musl] '@rollup/rollup-win32-arm64-msvc@4.45.0': resolution: {integrity: sha512-B+IJgcBnE2bm93jEW5kHisqvPITs4ddLOROAcOc/diBgrEiQJJ6Qcjby75rFSmH5eMGrqJryUgJDhrfj942apQ==} @@ -7329,24 +7418,28 @@ packages: engines: {node: '>=10'} cpu: [arm64] os: [linux] + libc: [glibc] '@swc/core-linux-arm64-musl@1.4.6': resolution: {integrity: sha512-LGQsKJ8MA9zZ8xHCkbGkcPSmpkZL2O7drvwsGKynyCttHhpwVjj9lguhD4DWU3+FWIsjvho5Vu0Ggei8OYi/Lw==} engines: {node: '>=10'} cpu: [arm64] os: [linux] + libc: [musl] '@swc/core-linux-x64-gnu@1.4.6': resolution: {integrity: sha512-10JL2nLIreMQDKvq2TECnQe5fCuoqBHu1yW8aChqgHUyg9d7gfZX/kppUsuimqcgRBnS0AjTDAA+JF6UsG/2Yg==} engines: {node: '>=10'} cpu: [x64] os: [linux] + libc: [glibc] '@swc/core-linux-x64-musl@1.4.6': resolution: {integrity: sha512-EGyjFVzVY6Do89x8sfah7I3cuP4MwtwzmA6OlfD/KASqfCFf5eIaEBMbajgR41bVfMV7lK72lwAIea5xEyq1AQ==} engines: {node: '>=10'} cpu: [x64] os: [linux] + libc: [musl] '@swc/core-win32-arm64-msvc@1.4.6': resolution: {integrity: sha512-gfW9AuXvwSyK07Vb8Y8E9m2oJZk21WqcD+X4BZhkbKB0TCZK0zk1j/HpS2UFlr1JB2zPKPpSWLU3ll0GEHRG2A==} @@ -8326,6 +8419,10 @@ packages: resolution: {integrity: sha512-AxNRwEie8Nn4eFS1FzDMJWIISMGoXMb037sgCBJ3UR6o0fQTzr2tqN9WT+DkWJPhIdQCfV7T6D387566VtnCJA==} engines: {node: ^18.18.0 || ^20.9.0 || >=21.1.0} + '@typespec/ts-http-runtime@0.3.3': + resolution: {integrity: sha512-91fp6CAAJSRtH5ja95T1FHSKa8aPW9/Zw6cta81jlZTUw/+Vq8jM/AfF/14h2b71wwR84JUTW/3Y8QPhDAawFA==} + engines: {node: '>=20.0.0'} + '@uiw/codemirror-extensions-basic-setup@4.21.24': resolution: {integrity: sha512-TJYKlPxNAVJNclW1EGumhC7I02jpdMgBon4jZvb5Aju9+tUzS44IwORxUx8BD8ZtH2UHmYS+04rE3kLk/BtnCQ==} peerDependencies: @@ -8403,41 +8500,49 @@ packages: resolution: {integrity: sha512-34gw7PjDGB9JgePJEmhEqBhWvCiiWCuXsL9hYphDF7crW7UgI05gyBAi6MF58uGcMOiOqSJ2ybEeCvHcq0BCmQ==} cpu: [arm64] os: [linux] + libc: [glibc] '@unrs/resolver-binding-linux-arm64-musl@1.11.1': resolution: {integrity: sha512-RyMIx6Uf53hhOtJDIamSbTskA99sPHS96wxVE/bJtePJJtpdKGXO1wY90oRdXuYOGOTuqjT8ACccMc4K6QmT3w==} cpu: [arm64] os: [linux] + libc: [musl] '@unrs/resolver-binding-linux-ppc64-gnu@1.11.1': resolution: {integrity: sha512-D8Vae74A4/a+mZH0FbOkFJL9DSK2R6TFPC9M+jCWYia/q2einCubX10pecpDiTmkJVUH+y8K3BZClycD8nCShA==} cpu: [ppc64] os: [linux] + libc: [glibc] '@unrs/resolver-binding-linux-riscv64-gnu@1.11.1': resolution: {integrity: sha512-frxL4OrzOWVVsOc96+V3aqTIQl1O2TjgExV4EKgRY09AJ9leZpEg8Ak9phadbuX0BA4k8U5qtvMSQQGGmaJqcQ==} cpu: [riscv64] os: [linux] + libc: [glibc] '@unrs/resolver-binding-linux-riscv64-musl@1.11.1': resolution: {integrity: sha512-mJ5vuDaIZ+l/acv01sHoXfpnyrNKOk/3aDoEdLO/Xtn9HuZlDD6jKxHlkN8ZhWyLJsRBxfv9GYM2utQ1SChKew==} cpu: [riscv64] os: [linux] + libc: [musl] '@unrs/resolver-binding-linux-s390x-gnu@1.11.1': resolution: {integrity: sha512-kELo8ebBVtb9sA7rMe1Cph4QHreByhaZ2QEADd9NzIQsYNQpt9UkM9iqr2lhGr5afh885d/cB5QeTXSbZHTYPg==} cpu: [s390x] os: [linux] + libc: [glibc] '@unrs/resolver-binding-linux-x64-gnu@1.11.1': resolution: {integrity: sha512-C3ZAHugKgovV5YvAMsxhq0gtXuwESUKc5MhEtjBpLoHPLYM+iuwSj3lflFwK3DPm68660rZ7G8BMcwSro7hD5w==} cpu: [x64] os: [linux] + libc: [glibc] '@unrs/resolver-binding-linux-x64-musl@1.11.1': resolution: {integrity: sha512-rV0YSoyhK2nZ4vEswT/QwqzqQXw5I6CjoaYMOX0TqBlWhojUf8P94mvI7nuJTeaCkkds3QE4+zS8Ko+GdXuZtA==} cpu: [x64] os: [linux] + libc: [musl] '@unrs/resolver-binding-wasm32-wasi@1.11.1': resolution: {integrity: sha512-5u4RkfxJm+Ng7IWgkzi3qrFOvLvQYnPBmjmZQ8+szTK/b31fQCnleNl1GgEt7nIsZRIf5PLhPwT0WM+q45x/UQ==} @@ -9060,6 +9165,11 @@ packages: axobject-query@4.0.0: resolution: {integrity: sha512-+60uv1hiVFhHZeO+Lz0RYzsVHy5Wr1ayX0mwda9KPDVLNJgZ1T9Ny7VmFbLDzxsH0D87I86vgj3gFrjTJUYznw==} + azure-storage@2.10.7: + resolution: {integrity: sha512-4oeFGtn3Ziw/fGs/zkoIpKKtygnCVIcZwzJ7UQzKTxhkGQqVCByOFbYqMGYR3L+wOsunX9lNfD0jc51SQuKSSA==} + engines: {node: '>= 0.8.26'} + deprecated: 'Please note: newer packages @azure/storage-blob, @azure/storage-queue and @azure/storage-file are available as of November 2019 and @azure/data-tables is available as of June 2021. While the legacy azure-storage package will continue to receive critical bug fixes, we strongly encourage you to upgrade. Migration guide can be found: https://github.com/Azure/azure-sdk-for-js/blob/main/sdk/storage/MigrationGuide.md' + b4a@1.6.6: resolution: {integrity: sha512-5Tk1HLk6b6ctmjIkAcU/Ujv/1WqiDl0F0JdRCR80VsOcUlHcu7pWeWRlOqQLHfDEsVx9YH/aif5AG4ehoCtTmg==} @@ -9311,6 +9421,9 @@ packages: browser-process-hrtime@1.0.0: resolution: {integrity: sha512-9o5UecI3GhkpM6DrXr69PblIuWxPKk9Y0jHBRhdocZ2y7YECBFCsHm79Pr3OyR2AvjhDkabFJaDJMYRazHgsow==} + browserify-mime@1.2.9: + resolution: {integrity: sha512-uz+ItyJXBLb6wgon1ELEiVowJBEsy03PUWGRQU7cxxx9S+DW2hujPp+DaMYEOClRPzsn7NB99NtJ6pGnt8y+CQ==} + browserslist@4.23.0: resolution: {integrity: sha512-QW8HiM1shhT2GuzkvklfjcKDiWFXHOeFCIA/huJPwHsslwcydgk7X+z2zXpEijP98UCY7HbubZt5J2Zgvf0CaQ==} engines: {node: ^6 || ^7 || ^8 || ^9 || ^10 || ^11 || ^12 || >=13.7} @@ -9421,6 +9534,14 @@ packages: resolution: {integrity: sha512-GHTSNSYICQ7scH7sZ+M2rFopRoLh8t2bLSW6BbgrtLsahOIB5iyAVJf9GjWK3cYTDaMj4XdBpM1cA6pIS0Kv2w==} engines: {node: '>= 0.4'} + call-bind@1.0.8: + resolution: {integrity: sha512-oKlSFMcMwpUg2ednkhQ454wfWiU/ul3CkJe/PEHcTKuiX6RpbehUiFMXu13HalGZxfUwCQzZG747YXBn1im9ww==} + engines: {node: '>= 0.4'} + + call-bound@1.0.4: + resolution: {integrity: sha512-+ys997U96po4Kx/ABpBCqhA9EuxJaQWDQg7295H4hBphv3IZg0boBKuwYpt4YXp6MZ5AmZQnU/tyMTlRpaSejg==} + engines: {node: '>= 0.4'} + call-me-maybe@1.0.2: resolution: {integrity: sha512-HpX65o1Hnr9HH25ojC1YGs7HCQLq0GCOibSaWER0eNpgJ/Z1MZv2mTc7+xh6WOPxbRVcmgbv4hGU+uSQ/2xFZQ==} @@ -9569,12 +9690,14 @@ packages: engines: {node: '>= 10'} cpu: [arm64] os: [linux] + libc: [glibc] chromadb-js-bindings-linux-x64-gnu@1.1.1: resolution: {integrity: sha512-RcvBcECbUcFXlFM2httdGc+3wmkI76tD9iGS0H9npEhz4LyLvdXKBK8CZtw67hsHCH09KklKw4ITkSS85pHVbA==} engines: {node: '>= 10'} cpu: [x64] os: [linux] + libc: [glibc] chromadb-js-bindings-win32-x64-msvc@1.1.1: resolution: {integrity: sha512-296SxWNwsmvP+1Ggkl72norTFLOivoXhGu0t5mXNIbd7yd2UodntvAvG2cheTHDCL/ILbox4u+6KHNvRxHgm6A==} @@ -11630,6 +11753,10 @@ packages: for-each@0.3.3: resolution: {integrity: sha512-jqYfLp7mo9vIyQf8ykW2v7A+2N4QjeCeI5+Dz9XraiO1ign81wjiH7Fb9vSOWvQfNtmSa4H2RoQTrrXivdUZmw==} + for-each@0.3.5: + resolution: {integrity: sha512-dKx12eRCVIzqCxFGplyFKJMPvLEWgmNtUrpTiJIR5u97zEhRG8ySrtboPHZXx7daLxQVrl643cTzbab2tkQjxg==} + engines: {node: '>= 0.4'} + for-in@1.0.2: resolution: {integrity: sha512-7EwmXrOjyL+ChxMhmG5lnW9MPt1aIeZEwKhQzoBUdTV0N3zuwWDZYVJatDvZ2OyzPUvdIAZDsCetk3coyMfcnQ==} engines: {node: '>=0.10.0'} @@ -12094,6 +12221,15 @@ packages: engines: {node: '>=0.4.7'} hasBin: true + har-schema@2.0.0: + resolution: {integrity: sha512-Oqluz6zhGX8cyRaTQlFMPw80bSJVG2x/cFb8ZPhUILGgHka9SsokCCOQgpveePerqidZOrT14ipqfJb7ILcW5Q==} + engines: {node: '>=4'} + + har-validator@5.1.5: + resolution: {integrity: sha512-nmT2T0lljbxdQZfspsno9hgrG3Uir6Ks5afism62poxqBM6sDnMEuPmzTq8XN0OEwqKLLdh1jQI3qyE66Nzb3w==} + engines: {node: '>=6'} + deprecated: this library is no longer supported + harmony-reflect@1.6.2: resolution: {integrity: sha512-HIp/n38R9kQjDEziXyDTuW3vvoxxyxjxFzXLrBr18uB47GnSt+G9D29fqrpM5ZkspMcPICud3XsBJQ4Y2URg8g==} @@ -12142,6 +12278,10 @@ packages: resolution: {integrity: sha512-+QdH3jOmq9P8GfdjFg0eJudqx1FqU62NQJ4P16rOEHeRdl7ckgwn6uqQjzYE0ZoHVV/e5E2esuJ5Gl5+HUW19w==} engines: {node: '>=6'} + hash-base@3.1.2: + resolution: {integrity: sha512-Bb33KbowVTIj5s7Ked1OsqHUeCpz//tPwR+E2zJgJKo9Z5XolZ9b6bdUgjmYlwnWhoOQKoTd1TYToZGn5mAYOg==} + engines: {node: '>= 0.8'} + hasown@2.0.2: resolution: {integrity: sha512-0hJU9SCPvmMzIBdZFqNPXWa6dqh7WdH0cII9y+CyS8rG3nL48Bclra9HmKhVVUHyPWNH5Y7xDwAB7bfgSjkUMQ==} engines: {node: '>= 0.4'} @@ -12329,6 +12469,10 @@ packages: resolution: {integrity: sha512-7mz/721AbnJwIVbnaSv1Cz3Am0ZLT/UBwkC92VlxhXv/k/BBQfM2fXElQNC27BVGr0uwUpplYPQM9LnaBMR5NQ==} engines: {node: '>=8.0.0'} + http-signature@1.2.0: + resolution: {integrity: sha512-CAbnr6Rz4CYQkLYUtSNXxQPUH2gK8f3iWexVlsnMeD+GjlsQ0Xsy1cOX+mN3dtxYomRy21CiOzU8Uhw6OwncEQ==} + engines: {node: '>=0.8', npm: '>=1.3.7'} + http-signature@1.3.6: resolution: {integrity: sha512-3adrsD6zqo4GsTqtO7FyrejHNv+NgiIfAfv68+jVlFmSr9OGy7zrxONceFRLKvnnZA5jbxQBX1u9PpB6Wi32Gw==} engines: {node: '>=0.10'} @@ -12784,6 +12928,10 @@ packages: resolution: {integrity: sha512-uZ25/bUAlUY5fR4OKT4rZQEBrzQWYV9ZJYGGsUmEJ6thodVJ1HX64ePQ6Z0qPWP+m+Uq6e9UugrE38jeYsDSMw==} engines: {node: '>= 0.4'} + is-typed-array@1.1.15: + resolution: {integrity: sha512-p3EcsicXjit7SaskXHs1hA91QxgTw46Fv6EFKKGS5DRFLD8yKnohjF3hxoju94b/OcMZoQukzpPpBE9uLVKzgQ==} + engines: {node: '>= 0.4'} + is-typedarray@1.0.0: resolution: {integrity: sha512-cyA56iCMHAh5CdzjJIa4aohJyeO1YbwLi3Jc35MmRU6poroFjIGZzUzupGiRPOjgHg9TLu43xbpwXk523fMxKA==} @@ -13258,6 +13406,9 @@ packages: json-buffer@3.0.1: resolution: {integrity: sha512-4bV5BfR2mqfQTJm+V5tPPdf+ZpuhiIvTuAB5g8kcrXOZpTT/QwwVRWBywX1ozr6lEuPdbHxwaJlm9G6mI2sfSQ==} + json-edm-parser@0.1.2: + resolution: {integrity: sha512-J1U9mk6lf8dPULcaMwALXB6yel3cJyyhk9Z8FQ4sMwiazNwjaUhegIcpZyZFNMvLRtnXwh+TkCjX9uYUObBBYA==} + json-parse-better-errors@1.0.2: resolution: {integrity: sha512-mrqyZKfX5EhL7hvqcV6WG1yYjnjeuYDzDhhcAAUrq8Po85NBQBJP+ZDUT75qZQ98IkUoBqdkExkukOU7Ts2wrw==} @@ -13302,6 +13453,10 @@ packages: jsonfile@6.1.0: resolution: {integrity: sha512-5dgndWOriYSm5cnYaJNhalLNDKOqFwyDB/rr1E9ZsGciGvKPs8R2xYGCacuf3z6K1YKDz182fd+fY3cn3pMqXQ==} + jsonparse@1.2.0: + resolution: {integrity: sha512-LkDEYtKnPFI9hQ/IURETe6F1dUH80cbRkaF6RaViSwoSNPwaxQpi6TgJGvJKyLQ2/9pQW+XCxK3hBoR44RAjkg==} + engines: {'0': node >= 0.2.0} + jsonpath@1.1.1: resolution: {integrity: sha512-l6Cg7jRpixfbgoWgkrl77dgEj8RPvND0wMH6TwQmi9Qs4TFfS9u5cUFnbeKTwj5ga5Y3BTGGNI28k117LJ009w==} @@ -13317,6 +13472,10 @@ packages: resolution: {integrity: sha512-PRp66vJ865SSqOlgqS8hujT5U4AOgMfhrwYIuIhfKaoSCZcirrmASQr8CX7cUg+RMih+hgznrjp99o+W4pJLHQ==} engines: {node: '>=12', npm: '>=6'} + jsprim@1.4.2: + resolution: {integrity: sha512-P2bSOMAc/ciLz6DzgjVlGJP9+BrJWu5UDGK70C2iweC5QBIeFf0ZXRvGjEj2uYgrY2MkAAhsSWHDWlFtEroZWw==} + engines: {node: '>=0.6.0'} + jsprim@2.0.2: resolution: {integrity: sha512-gqXddjPqQ6G40VdnI6T6yObEC+pDNvyP95wdQhkWkg7crHH3km5qP1FsOXEkzEQwnz6gz5qGTn1c2Y52wP3OyQ==} engines: {'0': node >=0.6.0} @@ -13873,6 +14032,9 @@ packages: md-utils-ts@2.0.0: resolution: {integrity: sha512-sMG6JtX0ebcRMHxYTcmgsh0/m6o8hGdQHFE2OgjvflRZlQM51CGGj/uuk056D+12BlCiW0aTpt/AdlDNtgQiew==} + md5.js@1.3.5: + resolution: {integrity: sha512-xitP+WxNPcTTOgnTJcrhM0xvdPepipPSf3I8EIpGKeFLjt3PlJLIDG3u8EX53ZIubkb+5U2+3rELYpEhHhzdkg==} + mdast-util-definitions@5.1.2: resolution: {integrity: sha512-8SVPMuHqlPME/z3gqVwWY4zVXn8lqKv/pAhC57FuJ40ImXyBpmO5ukh98zB2v7Blql2FiHjHv9LVztSIqjY+MA==} @@ -14365,6 +14527,9 @@ packages: muggle-string@0.3.1: resolution: {integrity: sha512-ckmWDJjphvd/FvZawgygcUeQCxzvohjFO5RxTjj4eq8kw359gFF3E1brjfI+viLMxss5JrHTDRHZvu2/tuy0Qg==} + multer-azure-blob-storage@1.2.0: + resolution: {integrity: sha512-jFxXMRqpeHPx4COOcgmLr4bd2qIGIRieanoEc9IcNZs5QPiZ+M4rD+21oUE8h3qdF5UyJwRbskZkYsmY3W+KMg==} + multer-cloud-storage@4.0.0: resolution: {integrity: sha512-BGY8mc0qjZYpjQKSBz18xOsOBJkujqN5CZs5F6Gwb7GwQSO19daHf/k7D63tND8wwX4xH3FeAVS3OJwexDmlkQ==} engines: {node: '>=14'} @@ -14673,6 +14838,9 @@ packages: nwsapi@2.2.7: resolution: {integrity: sha512-ub5E4+FBPKwAZx0UwIQOjYWGHTEq5sPqHQNRN8Z9e4A7u3Tj1weLJsL59yH9vmvqEtBHaOmT6cYQKIZOxp35FQ==} + oauth-sign@0.9.0: + resolution: {integrity: sha512-fexhUFFPTGV8ybAtSIGbV6gOkSv8UtRbDBnAyLQw4QPKkgNlsH2ByPGtMUqdWkos6YCRmAqViwgZrJc/mRDzZQ==} + oauth@0.10.0: resolution: {integrity: sha512-1orQ9MT1vHFGQxhuy7E/0gECD3fd2fCC+PIX+/jgmU/gI3EpRocXtmtvxCO5x3WZ443FLTLFWNDjl5MPJf9u+Q==} @@ -15098,6 +15266,9 @@ packages: resolution: {integrity: sha512-gDKb8aZMDeD/tZWs9P6+q0J9Mwkdl6xMV8TjnGP3qJVJ06bdMgkbBlLU8IdfOsIsFz2BW1rNVT3XuNEl8zPAvw==} engines: {node: '>=8'} + path@0.12.7: + resolution: {integrity: sha512-aXXC6s+1w7otVF9UletFkFcDsJeO7lSZBPUQhtb5O0xJe8LtYhj/GxldoL09bBj9+ZmE2hNoHqQSFMN5fikh4Q==} + pathe@1.1.2: resolution: {integrity: sha512-whLdWMYL2TwI08hn8/ZqAbrVemu0LNaNNJZX73O6qaIdCTfXutsLhMkjdENX0qhsQ9uIimo4/aQOmXkoon2nDQ==} @@ -16079,6 +16250,10 @@ packages: resolution: {integrity: sha512-+38qI9SOr8tfZ4QmJNplMUxqjbe7LKvvZgWdExBOmd+egZTtjLB67Gu0HRX3u/XOq7UU2Nx6nsjvS16Z9uwfpg==} engines: {node: '>=0.6'} + qs@6.5.5: + resolution: {integrity: sha512-mzR4sElr1bfCaPJe7m8ilJ6ZXdDaGoObcYR0ZHSsktM/Lt21MVHj5De30GQH2eiZ1qGRTO7LCAzQsUeXTNexWQ==} + engines: {node: '>=0.6'} + query-string@8.2.0: resolution: {integrity: sha512-tUZIw8J0CawM5wyGBiDOAp7ObdRQh4uBor/fUR9ZjmbZVvw95OD9If4w3MQxr99rg0DJZ/9CIORcpEqU5hQG7g==} engines: {node: '>=14.16'} @@ -16543,6 +16718,11 @@ packages: request-progress@3.0.0: resolution: {integrity: sha512-MnWzEHHaxHO2iWiQuHrUPBi/1WeBf5PkxQqNyNvLl9VAYSdXkP8tQ3pBSeCPD+yw0v0Aq1zosWLz0BdeXpWwZg==} + request@2.88.2: + resolution: {integrity: sha512-MsvtOrfG9ZcrOwAW+Qi+F6HbD0CWXEh9ou77uOb7FM2WPhwT7smM833PzanhJLsgXjN89Ir6V2PczXNnMpwKhw==} + engines: {node: '>= 6'} + deprecated: request has been deprecated, see https://github.com/request/request/issues/3142 + require-directory@2.1.1: resolution: {integrity: sha512-fGxEI7+wsG9xrvdjsrlmL22OMTTiHRwAMroiEeMgq8gzoLC/PQr7RsRDSTLUg/bZAZtF+TVIkHc6/4RIKrui+Q==} engines: {node: '>=0.10.0'} @@ -16796,6 +16976,9 @@ packages: engines: {node: '>=14.0.0'} hasBin: true + sax@0.5.8: + resolution: {integrity: sha512-c0YL9VcSfcdH3F1Qij9qpYJFpKFKMXNOkLWFssBL3RuF7ZS8oZhllR2rWlCRjDTJsfq3R6wbSsaRU6o0rkEdNw==} + sax@1.2.4: resolution: {integrity: sha512-NqVDv9TpANUjFm0N8uM5GxL36UgKi9/atZw+x7YFnQ8ckwFGKrl4xX4yWtrey3UJm5nP1kUbnYgLopqWNSRhWw==} @@ -17719,6 +17902,10 @@ packages: to-arraybuffer@1.0.1: resolution: {integrity: sha512-okFlQcoGTi4LQBG/PgSYblw9VOyptsz2KJZqc6qtgGdes8VktzUQkj4BI2blit072iS8VODNcMA+tvnS9dnuMA==} + to-buffer@1.2.2: + resolution: {integrity: sha512-db0E3UJjcFhpDhAF4tLo03oli3pwl3dbnzXOUIlRKrp+ldk/VUxzpWYZENsw2SZiuBjHAk7DfB0VU7NKdpb6sw==} + engines: {node: '>= 0.4'} + to-object-path@0.3.0: resolution: {integrity: sha512-9mWHdnGRuh3onocaHzukyvCZhzvr6tiflAy/JRFXcJX0TjgfWA9pk9t8CMbzmBE4Jfw58pXbkngtBtqYxzNEyg==} engines: {node: '>=0.10.0'} @@ -17750,6 +17937,10 @@ packages: resolution: {integrity: sha512-WBx8Uy5TLtOSRtIq+M03/sKDrXCLHxwDcquSP2c43Le03/9serjQBIztjRz6FkJez9D/hleyAXTBGLwwZUw9lA==} hasBin: true + tough-cookie@2.5.0: + resolution: {integrity: sha512-nlLsUzgm1kfLXSXfRZMc1KLAugd4hqJHDTvc2hDIwS3mZAfMEuMbc03SujMF+GEcpaX/qboeycw6iO8JwVv2+g==} + engines: {node: '>=0.8'} + tough-cookie@4.1.3: resolution: {integrity: sha512-aX/y5pVRkfRnfmuX+OdbSdXvPe6ieKX/G2s7e98f4poJHnqH3281gDPm/metm6E/WRamfx7WC4HUqkWHfQHprw==} engines: {node: '>=6'} @@ -17891,6 +18082,9 @@ packages: tslib@2.6.2: resolution: {integrity: sha512-AEYxH93jGFPn/a2iVAwW87VuUIkR1FVUKB77NwMF7nBTDkDrrT/Hpt/IrCJ0QXhW27jTBDcf5ZY7w6RiqTMw2Q==} + tslib@2.8.1: + resolution: {integrity: sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==} + tsutils@3.21.0: resolution: {integrity: sha512-mHKK3iUXL+3UF6xL5k0PEhKRUBKPBCv/+RkEOpjRWxxx27KKRBmmA60A9pgOUvMi8GKhRMPEmjBRPzs2W7O1OA==} engines: {node: '>= 6'} @@ -18002,6 +18196,10 @@ packages: resolution: {integrity: sha512-gEymJYKZtKXzzBzM4jqa9w6Q1Jjm7x2d+sh19AdsD4wqnMPDYyvwpsIc2Q/835kHuo3BEQ7CjelGhfTsoBb2MQ==} engines: {node: '>= 0.4'} + typed-array-buffer@1.0.3: + resolution: {integrity: sha512-nAYYwfY3qnzX30IkA6AQZjVbtK6duGontcQm1WSG1MD94YLqK0515GNApXkoxKOWMusVssAHWLh9SeaoefYFGw==} + engines: {node: '>= 0.4'} + typed-array-byte-length@1.0.1: resolution: {integrity: sha512-3iMJ9q0ao7WE9tWcaYKIptkNBuOIcZCCT0d4MRvuuH88fEoEH62IuQe0OtraD3ebQEoTRk8XCBoknUNc1Y67pw==} engines: {node: '>= 0.4'} @@ -18351,6 +18549,9 @@ packages: util.promisify@1.0.1: resolution: {integrity: sha512-g9JpC/3He3bm38zsLupWryXHoEcS22YHthuPQSJdMy6KNrzIRzWqcsHzD/WUnqe45whVou4VIsPew37DoXWNrA==} + util@0.10.4: + resolution: {integrity: sha512-0Pm9hTQ3se5ll1XihRic3FDIku70C+iHUdT/W926rSgHV5QgXsYbKZN8MSC3tJtSkhuROzvsQjAaFENRXr+19A==} + utila@0.4.0: resolution: {integrity: sha512-Z0DbgELS9/L/75wZbro8xAnT50pBVFQZ+hUEueGDU5FN51YSCYM+jdxsfCiHjwNP/4LCDD0i/graKpeBnOXKRA==} @@ -18366,6 +18567,11 @@ packages: resolution: {integrity: sha512-0/A9rDy9P7cJ+8w1c9WD9V//9Wj15Ce2MPz8Ri6032usz+NfePxx5AcN3bN+r6ZL6jEo066/yNYB3tn4pQEx+A==} hasBin: true + uuid@3.4.0: + resolution: {integrity: sha512-HjSDRw6gZE5JMggctHBcjVak08+KEVhSIiDzFnT9S9aegmp85S/bReBVTb4QTFaRNptJ9kuYaNhnbNEOkbKb/A==} + deprecated: Please upgrade to version 7 or higher. Older versions may use Math.random() in certain circumstances, which is known to be problematic. See https://v8.dev/blog/math-random for details. + hasBin: true + uuid@8.3.2: resolution: {integrity: sha512-+NYs2QeMWy+GWFOEm9xnn6HCDp0l7QBD7ml8zLUmJ+93Q5NF0NocErnwkTkXVFNiX3/fpC6afS8Dhb/gz7R7eg==} hasBin: true @@ -18754,6 +18960,10 @@ packages: resolution: {integrity: sha512-oV0jmFtUky6CXfkqehVvBP/LSWJ2sy4vWMioiENyJLePrBO/yKyV9OyJySfAKosh+RYkIl5zJCNZ8/4JncrpdA==} engines: {node: '>= 0.4'} + which-typed-array@1.1.20: + resolution: {integrity: sha512-LYfpUkmqwl0h9A2HL09Mms427Q1RZWuOHsukfVcKRq9q95iQxdw0ix1JQrqbcDR9PH1QDwf5Qo8OZb5lksZ8Xg==} + engines: {node: '>= 0.4'} + which@1.3.1: resolution: {integrity: sha512-HxJdYWq1MTIQbJ3nw0cqssHoTNU267KlrDuGZ1WYlxDStUtKUhOaJmh112/TZmHxxUfuJqPXSOm7tDyas0OSIQ==} hasBin: true @@ -18784,6 +18994,13 @@ packages: wink-nlp@2.3.0: resolution: {integrity: sha512-NcMmlsJavRZgaV4dAjsOQPuXG4v3yLRRssEibfx41lhmwTTOCaQGW7czNC73bDKCq7q4vqGTjX3/MFhK3I76TA==} + winston-azure-blob@1.5.0: + resolution: {integrity: sha512-X5YaNrbPEwLh+WWWWCZIqk6u/5trsj5bol3JkQ3MzZ5xD1/hFdc6d5UMSa+puboPKT27Euo3giCHNCBfW5JQug==} + engines: {node: '>=6.0.0'} + peerDependencies: + async: ^3.2.0 + winston: '>=3.0.0' + winston-daily-rotate-file@5.0.0: resolution: {integrity: sha512-JDjiXXkM5qvwY06733vf09I2wnMXpZEhxEVOSPenZMii+g7pcDcTBt2MRugnoi8BwVSuCT2jfRXBUy+n1Zz/Yw==} engines: {node: '>=8'} @@ -18967,6 +19184,9 @@ packages: resolution: {integrity: sha512-ICP2e+jsHvAj2E2lIHxa5tjXRlKDJo4IdvPvCXbXQGdzSfmSpNVyIKMvoZHjDY9DP0zV17iI85o90vRFXNccRw==} engines: {node: '>=12'} + xml2js@0.2.8: + resolution: {integrity: sha512-ZHZBIAO55GHCn2jBYByVPHvHS+o3j8/a/qmpEe6kxO3cTnTCWC3Htq9RYJ5G4XMwMMClD2QkXA9SNdPadLyn3Q==} + xml2js@0.6.2: resolution: {integrity: sha512-T4rieHaC1EXcES0Kxxj4JWgaUQHDk+qwHcYOCFHfiwKz7tOVPLq7Hjq9dM1WCMhylqMEfP7hMcOIChvotiZegA==} engines: {node: '>=4.0.0'} @@ -18979,6 +19199,10 @@ packages: resolution: {integrity: sha512-fDlsI/kFEx7gLvbecc0/ohLG50fugQp8ryHzMTuW9vSa1GJ0XYWKnhsUx7oie3G98+r56aTQIUB4kht42R3JvA==} engines: {node: '>=4.0'} + xmlbuilder@9.0.7: + resolution: {integrity: sha512-7YXTQc3P2l9+0rjaUbLwMKRhtmwg1M1eDf6nag7urC7pIPYLD9W/jmzQ4ptRSUbodw5S0jfoGTflLemQibSpeQ==} + engines: {node: '>=4.0'} + xmlchars@2.2.0: resolution: {integrity: sha512-JZnDKK8B0RCDw84FNdDAIpZK+JuJw+s7Lz8nksI7SIuU3UXJJslUthsi+uWBUYOwPFwW7W7PRLRfUKpxjtjFCw==} @@ -21656,6 +21880,119 @@ snapshots: '@smithy/types': 4.3.2 tslib: 2.6.2 + '@azure/abort-controller@2.1.2': + dependencies: + tslib: 2.8.1 + + '@azure/core-auth@1.10.1': + dependencies: + '@azure/abort-controller': 2.1.2 + '@azure/core-util': 1.13.1 + tslib: 2.8.1 + transitivePeerDependencies: + - supports-color + + '@azure/core-client@1.10.1': + dependencies: + '@azure/abort-controller': 2.1.2 + '@azure/core-auth': 1.10.1 + '@azure/core-rest-pipeline': 1.22.2 + '@azure/core-tracing': 1.3.1 + '@azure/core-util': 1.13.1 + '@azure/logger': 1.3.0 + tslib: 2.8.1 + transitivePeerDependencies: + - supports-color + + '@azure/core-http-compat@2.3.2(@azure/core-client@1.10.1)(@azure/core-rest-pipeline@1.22.2)': + dependencies: + '@azure/abort-controller': 2.1.2 + '@azure/core-client': 1.10.1 + '@azure/core-rest-pipeline': 1.22.2 + + '@azure/core-lro@2.7.2': + dependencies: + '@azure/abort-controller': 2.1.2 + '@azure/core-util': 1.13.1 + '@azure/logger': 1.3.0 + tslib: 2.8.1 + transitivePeerDependencies: + - supports-color + + '@azure/core-paging@1.6.2': + dependencies: + tslib: 2.8.1 + + '@azure/core-rest-pipeline@1.22.2': + dependencies: + '@azure/abort-controller': 2.1.2 + '@azure/core-auth': 1.10.1 + '@azure/core-tracing': 1.3.1 + '@azure/core-util': 1.13.1 + '@azure/logger': 1.3.0 + '@typespec/ts-http-runtime': 0.3.3 + tslib: 2.8.1 + transitivePeerDependencies: + - supports-color + + '@azure/core-tracing@1.3.1': + dependencies: + tslib: 2.8.1 + + '@azure/core-util@1.13.1': + dependencies: + '@azure/abort-controller': 2.1.2 + '@typespec/ts-http-runtime': 0.3.3 + tslib: 2.8.1 + transitivePeerDependencies: + - supports-color + + '@azure/core-xml@1.5.0': + dependencies: + fast-xml-parser: 5.2.5 + tslib: 2.8.1 + + '@azure/logger@1.3.0': + dependencies: + '@typespec/ts-http-runtime': 0.3.3 + tslib: 2.8.1 + transitivePeerDependencies: + - supports-color + + '@azure/storage-blob@12.31.0': + dependencies: + '@azure/abort-controller': 2.1.2 + '@azure/core-auth': 1.10.1 + '@azure/core-client': 1.10.1 + '@azure/core-http-compat': 2.3.2(@azure/core-client@1.10.1)(@azure/core-rest-pipeline@1.22.2) + '@azure/core-lro': 2.7.2 + '@azure/core-paging': 1.6.2 + '@azure/core-rest-pipeline': 1.22.2 + '@azure/core-tracing': 1.3.1 + '@azure/core-util': 1.13.1 + '@azure/core-xml': 1.5.0 + '@azure/logger': 1.3.0 + '@azure/storage-common': 12.3.0(@azure/core-client@1.10.1) + events: 3.3.0 + tslib: 2.8.1 + transitivePeerDependencies: + - supports-color + + '@azure/storage-common@12.3.0(@azure/core-client@1.10.1)': + dependencies: + '@azure/abort-controller': 2.1.2 + '@azure/core-auth': 1.10.1 + '@azure/core-http-compat': 2.3.2(@azure/core-client@1.10.1)(@azure/core-rest-pipeline@1.22.2) + '@azure/core-rest-pipeline': 1.22.2 + '@azure/core-tracing': 1.3.1 + '@azure/core-util': 1.13.1 + '@azure/logger': 1.3.0 + events: 3.3.0 + tslib: 2.8.1 + transitivePeerDependencies: + - '@azure/core-client' + - supports-color + '@babel/code-frame@7.26.2': dependencies: '@babel/helper-validator-identifier': 7.25.9 @@ -23416,11 +23753,6 @@ snapshots: tslib: 2.6.2 optional: true - '@emnapi/runtime@1.3.1': - dependencies: - tslib: 2.6.2 - optional: true - '@emnapi/runtime@1.8.1': dependencies: tslib: 2.6.2 @@ -24029,7 +24361,7 @@ snapshots: '@img/sharp-wasm32@0.33.5': dependencies: - '@emnapi/runtime': 1.3.1 + '@emnapi/runtime': 1.8.1 optional: true '@img/sharp-win32-ia32@0.33.5': @@ -24768,7 +25100,7 @@ snapshots: - encoding - openai - '@langchain/community@0.3.49(d32382e756a310688ffd878bb799886f)': + '@langchain/community@0.3.49(96a2dea491375c22412401f1555fc2c2)': dependencies: '@browserbasehq/stagehand': 1.9.0(@playwright/test@1.49.1)(bufferutil@4.0.8)(deepmerge@4.3.1)(dotenv@16.4.5)(encoding@0.1.13)(openai@4.96.0(encoding@0.1.13)(ws@8.18.3(bufferutil@4.0.8)(utf-8-validate@6.0.4))(zod@3.22.4))(utf-8-validate@6.0.4)(zod@3.22.4) '@ibm-cloud/watsonx-ai': 1.2.0 @@ -24793,6 +25125,7 @@ snapshots: '@aws-sdk/client-kendra': 3.750.0 '@aws-sdk/client-s3': 3.844.0 '@aws-sdk/credential-provider-node': 3.529.1 + '@azure/storage-blob': 12.31.0 '@browserbasehq/sdk': 2.0.0(encoding@0.1.13) '@datastax/astra-db-ts': 1.5.0 '@elastic/elasticsearch': 8.12.2 @@ -25111,9 +25444,9 @@ snapshots: - encoding - supports-color - '@mem0/community@0.0.1(5cc11fc2ac52307d4013e43691e7b64a)': + '@mem0/community@0.0.1(5a79e038a0f9851c2336c9f7472dc8f5)': dependencies: - '@langchain/community': 0.3.49(d32382e756a310688ffd878bb799886f) + '@langchain/community': 0.3.49(96a2dea491375c22412401f1555fc2c2) '@langchain/core': 0.3.61(openai@4.96.0(encoding@0.1.13)(ws@8.18.3(bufferutil@4.0.8)(utf-8-validate@6.0.4))(zod@3.22.4)) axios: 1.12.0(debug@4.3.4) mem0ai: 2.1.16(@anthropic-ai/sdk@0.65.0(zod@3.22.4))(@google/genai@0.7.0(bufferutil@4.0.8)(encoding@0.1.13)(utf-8-validate@6.0.4))(@mistralai/mistralai@0.1.3(encoding@0.1.13))(@qdrant/js-client-rest@1.9.0(typescript@5.5.2))(@supabase/supabase-js@2.39.8(bufferutil@4.0.8)(utf-8-validate@6.0.4))(@types/jest@29.5.14)(@types/pg@8.11.2)(@types/sqlite3@3.1.11)(encoding@0.1.13)(groq-sdk@0.5.0(encoding@0.1.13))(neo4j-driver@5.27.0)(ollama@0.5.11)(pg@8.11.3)(redis@4.6.13)(sqlite3@5.1.7)(ws@8.18.3(bufferutil@4.0.8)(utf-8-validate@6.0.4)) @@ -29372,6 +29705,14 @@ snapshots: '@typescript-eslint/types': 8.55.0 eslint-visitor-keys: 4.2.1 + '@typespec/ts-http-runtime@0.3.3': + dependencies: + http-proxy-agent: 7.0.2 + https-proxy-agent: 7.0.4 + tslib: 2.8.1 + transitivePeerDependencies: + - supports-color + '@uiw/codemirror-extensions-basic-setup@4.21.24(@codemirror/autocomplete@6.14.0(@codemirror/language@6.10.1)(@codemirror/state@6.4.1)(@codemirror/view@6.26.3)(@lezer/common@1.2.1))(@codemirror/commands@6.3.3)(@codemirror/language@6.10.1)(@codemirror/lint@6.5.0)(@codemirror/search@6.5.6)(@codemirror/state@6.4.1)(@codemirror/view@6.26.3)': dependencies: '@codemirror/autocomplete': 6.14.0(@codemirror/language@6.10.1)(@codemirror/state@6.4.1)(@codemirror/view@6.26.3)(@lezer/common@1.2.1) @@ -30207,6 +30548,21 @@ snapshots: dependencies: dequal: 2.0.3 + azure-storage@2.10.7: + dependencies: + browserify-mime: 1.2.9 + extend: 3.0.2 + json-edm-parser: 0.1.2 + json-schema: 0.4.0 + md5.js: 1.3.5 + readable-stream: 2.3.8 + request: 2.88.2 + underscore: 1.13.6 + uuid: 3.4.0 + validator: 13.12.0 + xml2js: 0.2.8 + xmlbuilder: 9.0.7 + b4a@1.6.6: {} babel-jest@27.5.1(@babel/core@7.24.0): @@ -30543,6 +30899,8 @@ snapshots: browser-process-hrtime@1.0.0: {} + browserify-mime@1.2.9: {} + browserslist@4.23.0: dependencies: caniuse-lite: 1.0.30001597 @@ -30688,6 +31046,18 @@ snapshots: get-intrinsic: 1.3.0 set-function-length: 1.2.2 + call-bind@1.0.8: + dependencies: + call-bind-apply-helpers: 1.0.2 + es-define-property: 1.0.1 + get-intrinsic: 1.3.0 + set-function-length: 1.2.2 + + call-bound@1.0.4: + dependencies: + call-bind-apply-helpers: 1.0.2 + get-intrinsic: 1.3.0 + call-me-maybe@1.0.2: {} callguard@2.0.0: {} @@ -33502,6 +33872,10 @@ snapshots: dependencies: is-callable: 1.2.7 + for-each@0.3.5: + dependencies: + is-callable: 1.2.7 + for-in@1.0.2: {} for-own@1.0.0: @@ -34151,6 +34525,13 @@ snapshots: optionalDependencies: uglify-js: 3.19.3 + har-schema@2.0.0: {} + + har-validator@5.1.5: + dependencies: + ajv: 6.12.6 + har-schema: 2.0.0 + harmony-reflect@1.6.2: {} has-bigints@1.0.2: {} @@ -34193,6 +34574,13 @@ snapshots: dependencies: kind-of: 6.0.3 + hash-base@3.1.2: + dependencies: + inherits: 2.0.4 + readable-stream: 2.3.8 + safe-buffer: 5.2.1 + to-buffer: 1.2.2 + hasown@2.0.2: dependencies: function-bind: 1.1.2 @@ -34476,6 +34864,12 @@ snapshots: transitivePeerDependencies: - debug + http-signature@1.2.0: + dependencies: + assert-plus: 1.0.0 + jsprim: 1.4.2 + sshpk: 1.18.0 + http-signature@1.3.6: dependencies: assert-plus: 1.0.0 @@ -34905,6 +35299,10 @@ snapshots: dependencies: which-typed-array: 1.1.15 + is-typed-array@1.1.15: + dependencies: + which-typed-array: 1.1.20 + is-typedarray@1.0.0: {} is-unc-path@1.0.0: @@ -35907,6 +36305,10 @@ snapshots: json-buffer@3.0.1: {} + json-edm-parser@0.1.2: + dependencies: + jsonparse: 1.2.0 + json-parse-better-errors@1.0.2: {} json-parse-even-better-errors@2.3.1: {} @@ -35948,6 +36350,8 @@ snapshots: optionalDependencies: graceful-fs: 4.2.11 + jsonparse@1.2.0: {} + jsonpath@1.1.1: dependencies: esprima: 1.2.2 @@ -35971,6 +36375,13 @@ snapshots: ms: 2.1.3 semver: 7.7.1 + jsprim@1.4.2: + dependencies: + assert-plus: 1.0.0 + extsprintf: 1.3.0 + json-schema: 0.4.0 + verror: 1.10.0 + jsprim@2.0.2: dependencies: assert-plus: 1.0.0 @@ -36644,6 +37055,12 @@ snapshots: md-utils-ts@2.0.0: {} + md5.js@1.3.5: + dependencies: + hash-base: 3.1.2 + inherits: 2.0.4 + safe-buffer: 5.2.1 + mdast-util-definitions@5.1.2: dependencies: '@types/mdast': 3.0.15 @@ -37291,6 +37708,16 @@ snapshots: muggle-string@0.3.1: {} + multer-azure-blob-storage@1.2.0: + dependencies: + azure-storage: 2.10.7 + express: 4.21.2 + multer: 1.4.5-lts.1 + path: 0.12.7 + uuid: 8.3.2 + transitivePeerDependencies: + - supports-color + multer-cloud-storage@4.0.0(encoding@0.1.13): dependencies: '@google-cloud/storage': 7.16.0(encoding@0.1.13) @@ -37638,6 +38065,8 @@ snapshots: nwsapi@2.2.7: {} + oauth-sign@0.9.0: {} + oauth@0.10.0: {} oauth@0.9.15: {} @@ -38163,6 +38592,11 @@ snapshots: path-type@4.0.0: {} + path@0.12.7: + dependencies: + process: 0.11.10 + util: 0.10.4 + pathe@1.1.2: {} pause-stream@0.0.11: @@ -39204,6 +39638,8 @@ snapshots: dependencies: side-channel: 1.0.6 + qs@6.5.5: {} + query-string@8.2.0: dependencies: decode-uri-component: 0.4.1 @@ -39901,6 +40337,29 @@ snapshots: dependencies: throttleit: 1.0.1 + request@2.88.2: + dependencies: + aws-sign2: 0.7.0 + aws4: 1.12.0 + caseless: 0.12.0 + combined-stream: 1.0.8 + extend: 3.0.2 + forever-agent: 0.6.1 + form-data: 4.0.4 + har-validator: 5.1.5 + http-signature: 1.2.0 + is-typedarray: 1.0.0 + isstream: 0.1.2 + json-stringify-safe: 5.0.1 + mime-types: 2.1.35 + oauth-sign: 0.9.0 + performance-now: 2.1.0 + qs: 6.5.5 + safe-buffer: 5.2.1 + tough-cookie: 2.5.0 + tunnel-agent: 0.6.0 + uuid: 3.4.0 + require-directory@2.1.1: {} require-from-string@2.0.2: {} @@ -40161,6 +40620,8 @@ snapshots: immutable: 4.3.5 source-map-js: 1.0.2 + sax@0.5.8: {} + sax@1.2.4: {} saxes@5.0.1: @@ -41354,6 +41815,12 @@ snapshots: to-arraybuffer@1.0.1: {} + to-buffer@1.2.2: + dependencies: + isarray: 2.0.5 + safe-buffer: 5.2.1 + typed-array-buffer: 1.0.3 + to-object-path@0.3.0: dependencies: kind-of: 3.2.2 @@ -41386,6 +41853,11 @@ snapshots: dependencies: nopt: 1.0.10 + tough-cookie@2.5.0: + dependencies: + psl: 1.9.0 + punycode: 2.3.1 + tough-cookie@4.1.3: dependencies: psl: 1.9.0 @@ -41515,6 +41987,8 @@ snapshots: tslib@2.6.2: {} + tslib@2.8.1: {} + tsutils@3.21.0(typescript@5.5.2): dependencies: tslib: 1.14.1 @@ -41604,6 +42078,12 @@ snapshots: es-errors: 1.3.0 is-typed-array: 1.1.13 + typed-array-buffer@1.0.3: + dependencies: + call-bound: 1.0.4 + es-errors: 1.3.0 + is-typed-array: 1.1.15 + typed-array-byte-length@1.0.1: dependencies: call-bind: 1.0.7 @@ -41998,6 +42478,10 @@ snapshots: has-symbols: 1.1.0 object.getownpropertydescriptors: 2.1.7 + util@0.10.4: + dependencies: + inherits: 2.0.3 + utila@0.4.0: {} utils-merge@1.0.1: {} @@ -42006,6 +42490,8 @@ snapshots: uuid@11.1.0: {} + uuid@3.4.0: {} + uuid@8.3.2: {} uuid@9.0.1: {} @@ -42516,6 +43002,16 @@ snapshots: gopd: 1.2.0 has-tostringtag: 1.0.2 + which-typed-array@1.1.20: + dependencies: + available-typed-arrays: 1.0.7 + call-bind: 1.0.8 + call-bound: 1.0.4 + for-each: 0.3.5 + get-proto: 1.0.1 + gopd: 1.2.0 + has-tostringtag: 1.0.2 + which@1.3.1: dependencies: isexe: 2.0.0 @@ -42547,6 +43043,18 @@ snapshots: wink-nlp@2.3.0: {} + winston-azure-blob@1.5.0(async@3.2.5)(winston@3.12.0): + dependencies: + '@azure/storage-blob': 12.31.0 + async: 3.2.5 + debug: 4.4.3(supports-color@8.1.1) + moment: 2.30.1 + triple-beam: 1.4.1 + winston: 3.12.0 + winston-transport: 4.7.0 + transitivePeerDependencies: + - supports-color + winston-daily-rotate-file@5.0.0(winston@3.12.0): dependencies: file-stream-rotator: 0.6.1 @@ -42870,6 +43378,10 @@ snapshots: xml-name-validator@4.0.0: {} + xml2js@0.2.8: + dependencies: + sax: 0.5.8 + xml2js@0.6.2: dependencies: sax: 1.2.4 @@ -42879,6 +43391,8 @@ snapshots: xmlbuilder@11.0.1: {} + xmlbuilder@9.0.7: {} + xmlchars@2.2.0: {} xmldom-sre@0.1.31: {}