mirror of
https://github.com/langgenius/dify-plugin-daemon.git
synced 2026-07-21 17:25:23 -04:00
S3 issue when using MinIO #141
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @DavideDelbianco on GitHub (Jun 11, 2025).
Originally assigned to: @41tair on GitHub.
Self Checks
To make sure we get to you in time, please check the following :)
Versions
Describe the bug
I have a local docker-compose deployment.
With previous version, I could use minio as local S3 storage and pass its address as http://minio:9000
With current version it no longer works.
It seems like the new dify-cloud-kit ignores the setting: S3_USE_PATH_STYLE and always uses the Virtualhosted style
Expected behavior
S3 storage should function just like before, and the PATH_STYLE variable should be respected
Screenshots
Additional context
My ENV variables for the demon config are those:
PLUGIN_STORAGE_TYPE: ${PLUGIN_STORAGE_TYPE:-aws_s3}
PLUGIN_STORAGE_OSS_BUCKET: ${PLUGIN_STORAGE_OSS_BUCKET:-dify-plugin}
S3_USE_AWS_MANAGED_IAM: ${PLUGIN_S3_USE_AWS_MANAGED_IAM:-false}
S3_USE_AWS_S3: ${PLUGIN_S3_USE_AWS_S3:-false}
S3_USE_PATH_STYLE: ${PLUGIN_S3_USE_PATH_STYLE:-true}
S3_ENDPOINT: ${PLUGIN_S3_ENDPOINT:-http://minio:9000}
AWS_ACCESS_KEY: ${PLUGIN_S3_ACCESS_KEY:-minio}
AWS_SECRET_KEY: ${PLUGIN_S3_SECRET_KEY:-miniosecret}
AWS_REGION: ${PLUGIN_S3_REGION:-auto}
@DavideDelbianco commented on GitHub (Jun 11, 2025):
I figured out the error.
I don't know why the new ENV variable was called: USE_AWS_S3 when every other S3 related variables starts with S3_
But I have erroneously passed the value "false" using S3_USE_AWS_S3 and the dify-cloud-kit uses the value of S3_USE_PATH_STYLE only when not using AWS client.
I will leave this issue open, because PATH_STYLE should be supported also in AWS S3
@41tair commented on GitHub (Jun 11, 2025):
From the screenshot, it appears that a service name is being used for the S3 endpoint, but MinIO is not in the same network, which could cause connectivity issues. Also, USE_AWS_S3 is currently used as a toggle to determine whether to use the AWS S3 API. Since all other related environment variables follow the S3_ prefix convention, do you have any suggestions for renaming it to stay consistent? Lastly, I’ll go ahead and fix the PATH_STYLE issue you mentioned.
@DavideDelbianco commented on GitHub (Jun 11, 2025):
"dify-plugin" Is the OSS bucket name.
When using S3 in virtual mode the url is
bucket-name.s3host.com
When it is in folder mode the url is:
s3host.com/bucket-name
This is why it is unable to resolve the url, virtual path creates a new domain level that in docker-compose cannot be resolved without the help of a reverse proxy like nginx, and even then MinIO wouldn't have supported it locally.
For the var name I would have just called it: S3_USE_AWS, Keeps the prefix S3, and uses the same boolean name used in the cloud-kit library.
It wouldn't have happened if the new ENV var was added to the docker compose and env.example file in the main project.
The default for the old IAM var was false, this one is true and completely flips the logic of the S3 storage.
It is a breaking change for whoever used an S3 storage that's not AWS
@41tair commented on GitHub (Jun 11, 2025):
You are right.
@DavideDelbianco commented on GitHub (Jun 11, 2025):
I've update my comment, please consider fixing the breaking change too.
Even if you handle the folder path style, the AWS S3 client handles differently the region.
Should have been false by default.
@41tair commented on GitHub (Jun 11, 2025):
I've updated the implementation based on your suggestion.
To clarify why the new environment variable wasn’t added to the main repository from the beginning:
In the dify repository, the S3_USE_AWS_MANAGED_IAM variable was originally intended to indicate whether IAM roles should be used for authorization when accessing S3. Since boto3 handles compatibility between AWS and other S3-compatible storage quite well, this worked without issue there.
However, in the early implementation of plugin-daemon, we used the AWS Go SDK, and mistakenly reused the S3_USE_AWS_MANAGED_IAM variable to determine whether the storage was AWS S3 or not. This led to incorrect logic and assumptions.
I've now introduced a separate variable to make this distinction explicit and will also submit a PR to propagate the new variable to the main repository.
@41tair commented on GitHub (Jun 13, 2025):
We have re-evaluated the impact of different environment variables, as illustrated in the diagram below. The conclusion is to continue using true as the S3_USE_AWS default value in order to minimize disruptions. However, this constitutes a breaking change for users of non-AWS storage, so we have included a special note about this scenario in the README.
