langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 20:15:31 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Unstructured API Calls when Dify selected as ETL #6809

New Issue
Closed
opened 2026-02-21 18:17:25 -05:00 by yindo · 6 comments
yindo commented 2026-02-21 18:17:25 -05:00
Owner
Copy Link
Copy Source

Originally created by @nsvrana on GitHub (Nov 17, 2024).

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

0.11.1

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

Self-hosting using Docker Compose, ETL_TYPE set to "dify." No uploaded knowledge, no active flows.

✔️ Expected Behavior

Nothing - I wouldn't expect Dify to interact with the unstructured.io URL at all.

❌ Actual Behavior

Every 12-13 seconds there are two DNS lookups up from the server to unstructured.io per my local DNS server. Continues whether you set an unstructured API url or not.

image

Originally created by @nsvrana on GitHub (Nov 17, 2024). ### Self Checks - [X] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [X] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [X] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [X] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:） - [X] Please do not modify this template :) and fill in all the required fields. ### Dify version 0.11.1 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce Self-hosting using Docker Compose, ETL_TYPE set to "dify." No uploaded knowledge, no active flows. ### ✔️ Expected Behavior Nothing - I wouldn't expect Dify to interact with the unstructured.io URL at all. ### ❌ Actual Behavior Every 12-13 seconds there are two DNS lookups up from the server to unstructured.io per my local DNS server. Continues whether you set an unstructured API url or not. ![image](https://github.com/user-attachments/assets/31444785-5fc4-409c-b1c4-51e4110dc048)
yindo added the 🐞 bug label 2026-02-21 18:17:25 -05:00
yindo closed this issue 2026-02-21 18:17:25 -05:00
yindo commented 2026-02-21 18:17:25 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Nov 17, 2024):

I couldn't find specific information in the available data about why DNS lookups to unstructured.io occur when ETL_TYPE is set to "dify" in Dify version 0.11.1. This behavior might be due to a configuration issue or a bug in the application. It would be beneficial to check the Dify GitHub repository for any related issues or updates that might address this unexpected behavior.

To continue talking to Dosu, mention @dosu.

@dosubot[bot] commented on GitHub (Nov 17, 2024): <!-- Answer --> I couldn't find specific information in the available data about why DNS lookups to unstructured.io occur when `ETL_TYPE` is set to "dify" in Dify version 0.11.1. This behavior might be due to a configuration issue or a bug in the application. It would be beneficial to check the Dify GitHub repository for any related issues or updates that might address this unexpected behavior. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
yindo commented 2026-02-21 18:17:25 -05:00
Author
Owner
Copy Link
Copy Source

@nsvrana commented on GitHub (Nov 17, 2024):

Checking the logs, it looks like my worker is crashing, and when it reboots, it calls unstructured.io. Why does the worker make an unstructured.io call when unstructured is unused?

@nsvrana commented on GitHub (Nov 17, 2024): Checking the logs, it looks like my worker is crashing, and when it reboots, it calls unstructured.io. Why does the worker make an unstructured.io call when unstructured is unused?
yindo commented 2026-02-21 18:17:25 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Nov 17, 2024):

The Dify codebase includes logging and error handling that might help diagnose why the worker is making calls to unstructured.io. The AdvancedChatAppGenerator class logs exceptions during the generation process, which could provide insights into the worker's behavior. Additionally, Dify integrates with Sentry for error tracking, which can capture and report errors, potentially shedding light on the reasons behind the worker's actions, including any calls to external services like unstructured.io [1][2].

If the worker is crashing and then making these calls upon reboot, it might be worth investigating the logs and Sentry reports for any errors or exceptions that occur just before or after the crash. This could help identify if there's a specific part of the code or a configuration that inadvertently triggers these calls.

To continue talking to Dosu, mention @dosu.

@dosubot[bot] commented on GitHub (Nov 17, 2024): <!-- Answer --> The Dify codebase includes logging and error handling that might help diagnose why the worker is making calls to unstructured.io. The `AdvancedChatAppGenerator` class logs exceptions during the generation process, which could provide insights into the worker's behavior. Additionally, Dify integrates with Sentry for error tracking, which can capture and report errors, potentially shedding light on the reasons behind the worker's actions, including any calls to external services like unstructured.io <sup>[[1]](https://github.com/langgenius/dify/blob/main/api/core/app/apps/advanced_chat/app_generator.py)</sup><sup>[[2]](https://github.com/langgenius/dify/blob/main/api/extensions/ext_sentry.py)</sup>. If the worker is crashing and then making these calls upon reboot, it might be worth investigating the logs and Sentry reports for any errors or exceptions that occur just before or after the crash. This could help identify if there's a specific part of the code or a configuration that inadvertently triggers these calls. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
yindo commented 2026-02-21 18:17:25 -05:00
Author
Owner
Copy Link
Copy Source

@zolgear commented on GitHub (Nov 27, 2024):

This is because the unstructured.io package enables telemetry by default.

Even when self-hosting with ETL_TYPE=Unstructured, it will make network requests.

There is an environment variable to disable telemetry. Consider adding it to the API configuration:

  environment:
    SCARF_NO_ANALYTICS: true

Ref: https://docs.unstructured.io/api-reference/api-services/free-api#telemetry

@zolgear commented on GitHub (Nov 27, 2024): This is because the `unstructured.io` package enables telemetry by default. Even when self-hosting with `ETL_TYPE=Unstructured`, it will make network requests. There is an environment variable to disable telemetry. Consider adding it to the API configuration: ``` environment: SCARF_NO_ANALYTICS: true ``` Ref: <https://docs.unstructured.io/api-reference/api-services/free-api#telemetry>
yindo commented 2026-02-21 18:17:26 -05:00
Author
Owner
Copy Link
Copy Source

@zolgear commented on GitHub (Dec 17, 2024):

ref #11179
Fixed.

@zolgear commented on GitHub (Dec 17, 2024): ref #11179 Fixed.
yindo commented 2026-02-21 18:17:26 -05:00
Author
Owner
Copy Link
Copy Source

@nsvrana commented on GitHub (Dec 17, 2024):

Thanks Zolgear

@nsvrana commented on GitHub (Dec 17, 2024): Thanks Zolgear
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#6809
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?