langgenius/dify-plugin-daemon
1
0
Fork 0
mirror of https://github.com/langgenius/dify-plugin-daemon.git synced 2026-07-21 17:25:23 -04:00
Code Issues 111 Packages Projects Releases 10 Wiki Activity

[PR #452] Feat: Add CLI to migrate plugin storage from local to cloud #513

New Issue
Open
opened 2026-02-16 01:16:11 -05:00 by yindo · 0 comments
yindo commented 2026-02-16 01:16:11 -05:00
Owner
Copy Link
Copy Source

📋 Pull Request Information

Original PR: https://github.com/langgenius/dify-plugin-daemon/pull/452
Author: @quicksandznzn
Created: 9/19/2025
Status: 🔄 Open

Base: main ← Head: feat/storage-migration

📝 Commits (3)

  • 535ebeb Feat: Add CLI to migrate plugin storage from local to cloud
  • a1aa2fc retrigger CI
  • b921a13 storagefactory

📊 Changes

6 files changed (+382 additions, -67 deletions)

View changed files

📝 README.md (+4 -0)
➕ cmd/migrate_storage/main.go (+189 -0)
➕ docs/storage-migration/guide.md (+53 -0)
➕ docs/storage-migration/guide_cn.md (+52 -0)
📝 internal/server/server.go (+14 -67)
➕ internal/storagefactory/factory.go (+70 -0)

📄 Description

Description

Fixes: #448

Summary

Add a CLI to migrate plugin storage from local filesystem to a cloud object store, plus bilingual docs and README links.

Motivation

  • Simplify moving existing local plugin data (packages, assets, installed bundles) to cloud storage.
  • Standardize on the same OSS abstraction used by the daemon.
  • Provide safe, idempotent migration with preview mode.

Key Changes

  • New CLI: cmd/migrate_storage/main.go
    • Recursively migrates three categories: plugin_packages, assets, plugin
    • Idempotent uploads: skips destination objects that already exist
    • Flags:
      • --dry-run (preview only)
      • --only packages,assets,installed (scope control)
      • --source-root (override local root)
    • Safety: restricts to local → cloud only (rejects local → local)
    • Fix: corrects prefix handling when listing/loading/saving under a given path
  • Documentation (bilingual):
    • English: docs/storage-migration/guide.md
    • 中文: docs/storage-migration/guide_cn.md
  • README:
    • New “Storage Migration” section with links to both guides

How To Use

  • Configure environment (or .env) with the same variables used by the daemon:
    • PLUGIN_STORAGE_TYPE (must be non-local)
    • PLUGIN_STORAGE_OSS_BUCKET
    • Cloud provider credentials (AWS/COS/OSS/Azure/GCS/OBS/TOS)
    • PLUGIN_STORAGE_LOCAL_ROOT (defaults to ./storage)
  • Run: 
    # Preview
go run ./cmd/migrate_storage --dry-run

# Execute
go run ./cmd/migrate_storage

# Or build and run
go build -o migrate-storage ./cmd/migrate_storage
./migrate-storage --only packages,assets,installed

Notes and Constraints

  • Only local → cloud is supported; the tool exits if PLUGIN_STORAGE_TYPE=local.
  • Requires network access and write permission to the destination bucket/container.
  • No database changes are made; does not alter install_type or installation records.

File References

  • cmd/migrate_storage/main.go
  • docs/storage-migration/guide.md
  • docs/storage-migration/guide_cn.md
  • README.md (adds “Storage Migration” section)

Known Limitations

  • For very large datasets, many head/list operations can be slower on some providers.
  • Migration operates single-process; concurrency tuning can be considered later if needed.

Type of Change

  • Bug fix
  • New feature
  • Refactor
  • Performance improvement
  • Other

Essential Checklist

Testing

  • I have tested the changes locally and confirmed they work as expected
  • I have added unit tests where necessary and they pass successfully

Bug Fix (if applicable)

  • I have used GitHub syntax to close the related issue (e.g., Fixes #123 or Closes #123)

Additional Information

1. Migration Command Screenshot

Migration Command

2. Cloud OSS Screenshot

Cloud OSS Configuration

3. Change Plugin Storage Type

Plugin Storage Type Configuration

4. Restart Plugin Daemon

I deleted the cwd folder to test the initialization of loading the plugin from the cloud. The plugin daemon runs successfully

Plugin Daemon Restart

5. Test the Plugin in Dify

Plugin Testing in Dify

Please provide any additional context that would help reviewers understand the changes.

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/langgenius/dify-plugin-daemon/pull/452 **Author:** [@quicksandznzn](https://github.com/quicksandznzn) **Created:** 9/19/2025 **Status:** 🔄 Open **Base:** `main` ← **Head:** `feat/storage-migration` --- ### 📝 Commits (3) - [`535ebeb`](https://github.com/langgenius/dify-plugin-daemon/commit/535ebebc7dfd21143a51a8bd23fffd45893a1d83) Feat: Add CLI to migrate plugin storage from local to cloud - [`a1aa2fc`](https://github.com/langgenius/dify-plugin-daemon/commit/a1aa2fcbd35050814bb54202aeaa6247dfcb2668) retrigger CI - [`b921a13`](https://github.com/langgenius/dify-plugin-daemon/commit/b921a13ecd52eb54fa5e135f3fa1fccbb3289d9c) storagefactory ### 📊 Changes **6 files changed** (+382 additions, -67 deletions) <details> <summary>View changed files</summary> 📝 `README.md` (+4 -0) ➕ `cmd/migrate_storage/main.go` (+189 -0) ➕ `docs/storage-migration/guide.md` (+53 -0) ➕ `docs/storage-migration/guide_cn.md` (+52 -0) 📝 `internal/server/server.go` (+14 -67) ➕ `internal/storagefactory/factory.go` (+70 -0) </details> ### 📄 Description ## Description Fixes: #448 ### Summary Add a CLI to migrate plugin storage from local filesystem to a cloud object store, plus bilingual docs and README links. ### Motivation - Simplify moving existing local plugin data (packages, assets, installed bundles) to cloud storage. - Standardize on the same OSS abstraction used by the daemon. - Provide safe, idempotent migration with preview mode. ### Key Changes - New CLI: `cmd/migrate_storage/main.go` - Recursively migrates three categories: `plugin_packages`, `assets`, `plugin` - Idempotent uploads: skips destination objects that already exist - Flags: - `--dry-run` (preview only) - `--only packages,assets,installed` (scope control) - `--source-root` (override local root) - Safety: restricts to local → cloud only (rejects local → local) - Fix: corrects prefix handling when listing/loading/saving under a given path - Documentation (bilingual): - English: `docs/storage-migration/guide.md` - 中文: `docs/storage-migration/guide_cn.md` - README: - New “Storage Migration” section with links to both guides ### How To Use - Configure environment (or `.env`) with the same variables used by the daemon: - `PLUGIN_STORAGE_TYPE` (must be non-`local`) - `PLUGIN_STORAGE_OSS_BUCKET` - Cloud provider credentials (AWS/COS/OSS/Azure/GCS/OBS/TOS) - `PLUGIN_STORAGE_LOCAL_ROOT` (defaults to `./storage`) - Run: ```bash # Preview go run ./cmd/migrate_storage --dry-run # Execute go run ./cmd/migrate_storage # Or build and run go build -o migrate-storage ./cmd/migrate_storage ./migrate-storage --only packages,assets,installed ``` ### Notes and Constraints - Only local → cloud is supported; the tool exits if `PLUGIN_STORAGE_TYPE=local`. - Requires network access and write permission to the destination bucket/container. - No database changes are made; does not alter `install_type` or installation records. ### File References - `cmd/migrate_storage/main.go` - `docs/storage-migration/guide.md` - `docs/storage-migration/guide_cn.md` - `README.md` (adds “Storage Migration” section) ### Known Limitations - For very large datasets, many head/list operations can be slower on some providers. - Migration operates single-process; concurrency tuning can be considered later if needed. ## Type of Change - [ ] Bug fix - [x] New feature - [ ] Refactor - [ ] Performance improvement - [ ] Other ## Essential Checklist ### Testing - [ ] I have tested the changes locally and confirmed they work as expected - [ ] I have added unit tests where necessary and they pass successfully ### Bug Fix (if applicable) - [ ] I have used GitHub syntax to close the related issue (e.g., `Fixes #123` or `Closes #123`) ## Additional Information ## 1. Migration Command Screenshot ![Migration Command](https://github.com/user-attachments/assets/a86795ba-ebf0-4374-9411-83cdf9d92847) ## 2. Cloud OSS Screenshot ![Cloud OSS Configuration](https://github.com/user-attachments/assets/814b908e-561a-4759-8cd7-06b871b62bd0) ## 3. Change Plugin Storage Type ![Plugin Storage Type Configuration](https://github.com/user-attachments/assets/f573a036-fbf7-4ae8-9333-04af850b61ed) ## 4. Restart Plugin Daemon **I deleted the cwd folder to test the initialization of loading the plugin from the cloud. The plugin daemon runs successfully** ![Plugin Daemon Restart](https://github.com/user-attachments/assets/2c9648ca-cba9-4d6b-99fb-d34517e034cf) ## 5. Test the Plugin in Dify ![Plugin Testing in Dify](https://github.com/user-attachments/assets/bbc09ad7-bc13-4124-9f41-916fee14b4ec) Please provide any additional context that would help reviewers understand the changes. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
yindo added the pull-request label 2026-02-16 01:16:11 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Windows
bug
cmd
documentation
enhancement
go
good first issue
plugin-daemon
pull-request
Mirrored from GitHub Pull Request
 question
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify-plugin-daemon#513
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?