langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 12:05:30 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Integrate Apache OpenDAL for Extended Storage Backend Support #6077

New Issue
Closed
opened 2026-02-21 18:13:58 -05:00 by yindo · 7 comments
yindo commented 2026-02-21 18:13:58 -05:00
Owner
Copy Link
Copy Source

Originally created by @laipz8200 on GitHub (Oct 14, 2024).

Originally assigned to: @laipz8200 on GitHub.

Self Checks

  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
  • Please do not modify this template :) and fill in all the required fields.

1. Is this request related to a challenge you're experiencing? Tell me about your story.

We propose integrating Apache OpenDAL into Dify to expand its support for various file storage backends and streamline configuration management. This integration will allow Dify to adopt a more flexible and unified approach for handling storage solutions.

Implementation Plan:

  1. Introduce a new OpenDAL adapter:
    Add a new adapter for OpenDAL to coexist alongside the existing storage solutions currently supported by Dify.

  2. Consolidate existing storage solutions:
    For storage backends already supported by OpenDAL, we plan to unify these into the OpenDAL framework, subsequently removing their standalone dependencies to reduce redundancy.

Benefits:

  • Expanded support for additional file storage backends.
  • Simplified and unified configuration management.
  • Reduced maintenance by relying on OpenDAL's ecosystem.
  • Better foundation for supporting future knowledge base or file source integrations.

2. Additional context or comments

No response

3. Can you help us with this feature?

  • I am interested in contributing to this feature.
Originally created by @laipz8200 on GitHub (Oct 14, 2024). Originally assigned to: @laipz8200 on GitHub. ### Self Checks - [X] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [X] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [X] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:） - [X] Please do not modify this template :) and fill in all the required fields. ### 1. Is this request related to a challenge you're experiencing? Tell me about your story. We propose integrating Apache OpenDAL into Dify to expand its support for various file storage backends and streamline configuration management. This integration will allow Dify to adopt a more flexible and unified approach for handling storage solutions. ### Implementation Plan: 1. **Introduce a new [OpenDAL](https://github.com/apache/opendal) adapter:** Add a new adapter for OpenDAL to coexist alongside the existing storage solutions currently supported by Dify. 2. **Consolidate existing storage solutions:** For storage backends already supported by OpenDAL, we plan to unify these into the OpenDAL framework, subsequently removing their standalone dependencies to reduce redundancy. ### Benefits: - Expanded support for additional file storage backends. - Simplified and unified configuration management. - Reduced maintenance by relying on OpenDAL's ecosystem. - Better foundation for supporting future knowledge base or file source integrations. ### 2. Additional context or comments _No response_ ### 3. Can you help us with this feature? - [X] I am interested in contributing to this feature.
yindo added the 💪 enhancement label 2026-02-21 18:13:58 -05:00
yindo closed this issue 2026-02-21 18:13:58 -05:00
yindo commented 2026-02-21 18:13:58 -05:00
Author
Owner
Copy Link
Copy Source

@laipz8200 commented on GitHub (Nov 14, 2024):

Will start after upgrading to Python 3.12

@laipz8200 commented on GitHub (Nov 14, 2024): Will start after upgrading to Python 3.12
yindo commented 2026-02-21 18:13:59 -05:00
Author
Owner
Copy Link
Copy Source

@Xuanwo commented on GitHub (Nov 16, 2024):

Hi, please let me know if you need any help! I believe @PsiACE will also be interested.

@Xuanwo commented on GitHub (Nov 16, 2024): Hi, please let me know if you need any help! I believe @PsiACE will also be interested.
yindo commented 2026-02-21 18:13:59 -05:00
Author
Owner
Copy Link
Copy Source

@Zheaoli commented on GitHub (Nov 17, 2024):

I'm the maintainer of the OpenDAL Python binding. Feel free to ask me if you get any issues when you use OpenDAL

@Zheaoli commented on GitHub (Nov 17, 2024): I'm the maintainer of the OpenDAL Python binding. Feel free to ask me if you get any issues when you use OpenDAL
yindo commented 2026-02-21 18:13:59 -05:00
Author
Owner
Copy Link
Copy Source

@PsiACE commented on GitHub (Nov 18, 2024):

I've just looked at it briefly haven't scrutinized it yet, but if it's from boto3 to opendal it should be easier overall, or ping me if you need help.

@PsiACE commented on GitHub (Nov 18, 2024): I've just looked at it briefly haven't scrutinized it yet, but if it's from boto3 to opendal it should be easier overall, or ping me if you need help.
yindo commented 2026-02-21 18:13:59 -05:00
Author
Owner
Copy Link
Copy Source

@laipz8200 commented on GitHub (Dec 10, 2024):

Hi @PsiACE, sorry to bother you! I’m currently working on migrating our S3 storage from boto3 to OpenDAL. We’ve been using an IAM-based connection method (as implemented in extensions/storage/aws_s3_storage.py, line 20). Does OpenDAL support this type of connection?

Thanks so much for your time!

@laipz8200 commented on GitHub (Dec 10, 2024): Hi @PsiACE, sorry to bother you! I’m currently working on migrating our S3 storage from boto3 to OpenDAL. We’ve been using an IAM-based connection method (as implemented in extensions/storage/aws_s3_storage.py, line 20). Does OpenDAL support this type of connection? Thanks so much for your time!
yindo commented 2026-02-21 18:13:59 -05:00
Author
Owner
Copy Link
Copy Source

@PsiACE commented on GitHub (Dec 10, 2024):

We’ve been using an IAM-based connection method (as implemented in extensions/storage/aws_s3_storage.py, line 20). Does OpenDAL support this type of connection?

Yes, OpenDAL supports IAM. For each specific service, there are some documentation/workflows available, for example:

@PsiACE commented on GitHub (Dec 10, 2024): > We’ve been using an IAM-based connection method (as implemented in extensions/storage/aws_s3_storage.py, line 20). Does OpenDAL support this type of connection? Yes, OpenDAL supports IAM. For each specific service, there are some documentation/workflows available, for example: - https://github.com/apache/opendal/blob/main/core/src/services/s3/docs.md - https://github.com/apache/opendal/blob/7cde1f585bc00e42c459a463956b6bc260d2eaaa/.github/workflows/test_edge.yml
yindo commented 2026-02-21 18:13:59 -05:00
Author
Owner
Copy Link
Copy Source

@laipz8200 commented on GitHub (Dec 12, 2024):

Thank you all for the help! We can release this feature in the next version now! ❤️

@laipz8200 commented on GitHub (Dec 12, 2024): Thank you all for the help! We can release this feature in the next version now! ❤️
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 💪 enhancement
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#6077
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?