dify_extractor: opt-out [email protected] #799

New Issue

Closed

opened 2026-02-16 10:20:32 -05:00 by yindo · 7 comments

yindo commented 2026-02-16 10:20:32 -05:00

Owner

Copy Link

Copy Source

Originally created by @davidkhala on GitHub (Nov 11, 2025).

Self Checks

• I have read the Contributing Guide and Language Policy.
• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report, otherwise it will be closed.
• Please do not modify this template :) and fill in all the required fields.

1. Is this request related to a challenge you're experiencing? Tell me about your story.

I am trying to parse uploaded html into chunk. The html has a general footer that containing public contact information (hotline, enquiry email).

I use File as data source, followed by Dify Extractor, surprisingly it mask all email with [email protected]. I believe it is interim security workaround. Thus I want to have freedom to configure it with a bool param, that I can keep raw email address

2. Additional context or comments

name: dify_extractor
version: 0.0.5

3. Can you help us with this feature?

• I am interested in contributing to this feature.

Originally created by @davidkhala on GitHub (Nov 11, 2025). ### Self Checks - [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify-official-plugins/issues), including closed ones. - [x] I confirm that I am using English to submit this report, otherwise it will be closed. - [x] Please do not modify this template :) and fill in all the required fields. ### 1. Is this request related to a challenge you're experiencing? Tell me about your story. I am trying to parse uploaded html into chunk. The html has a general footer that containing public contact information (hotline, enquiry email). I use File as data source, followed by Dify Extractor, surprisingly it mask all email with [email protected]. I believe it is interim security workaround. Thus I want to have freedom to configure it with a bool param, that I can keep raw email address ### 2. Additional context or comments name: dify_extractor version: 0.0.5 ### 3. Can you help us with this feature? - [x] I am interested in contributing to this feature.

yindo added the enhancement label 2026-02-16 10:20:32 -05:00

yindo closed this issue 2026-02-16 10:20:32 -05:00

yindo commented 2026-02-16 10:20:32 -05:00

Author

Owner

Copy Link

Copy Source

@dosubot[bot] commented on GitHub (Nov 11, 2025):

The dify_extractor plugin does not currently implement email masking or provide a configuration option for it. The HTML extraction logic simply returns raw text and does not alter email addresses, so the masking you observed ([email protected]) is likely performed by another part of the Dify system or a post-processing step outside this plugin (source).

If you want to add a configurable email masking feature to dify_extractor, the standard approach is to introduce a new boolean parameter in the plugin's YAML config (tools/dify_extractor/tools/dify_extractor.yaml) and update the extractor code to respect this parameter. There is currently no masking logic or config parameter for this in the codebase (source).

You may want to check upstream or workflow components in your Dify deployment to locate where the masking is actually performed, as changes may be needed there instead. If you decide to contribute this feature, following the YAML config pattern and updating the extractor logic is the recommended path.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}   

@dosubot[bot] commented on GitHub (Nov 11, 2025): <!-- Answer --> The dify_extractor plugin does not currently implement email masking or provide a configuration option for it. The HTML extraction logic simply returns raw text and does not alter email addresses, so the masking you observed ([email protected]) is likely performed by another part of the Dify system or a post-processing step outside this plugin ([source](https://github.com/langgenius/dify-official-plugins/blob/3eae206018c83c8ace10123e3fda3ee4f5926e0e/tools/dify_extractor/tools/html_extractor.py#L9-L41)). If you want to add a configurable email masking feature to dify_extractor, the standard approach is to introduce a new boolean parameter in the plugin's YAML config (`tools/dify_extractor/tools/dify_extractor.yaml`) and update the extractor code to respect this parameter. There is currently no masking logic or config parameter for this in the codebase ([source](https://github.com/langgenius/dify-official-plugins/blob/3eae206018c83c8ace10123e3fda3ee4f5926e0e/tools/dify_extractor/tools/dify_extractor.yaml#L14-L44)). You may want to check upstream or workflow components in your Dify deployment to locate where the masking is actually performed, as changes may be needed there instead. If you decide to contribute this feature, following the YAML config pattern and updating the extractor logic is the recommended path. <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- ^{How did I do? [Good](https://app.dosu.dev/response-feedback/064a02fa-e1fd-4c08-8f42-712e116a8290?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/064a02fa-e1fd-4c08-8f42-712e116a8290?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/064a02fa-e1fd-4c08-8f42-712e116a8290?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/064a02fa-e1fd-4c08-8f42-712e116a8290?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/064a02fa-e1fd-4c08-8f42-712e116a8290?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/064a02fa-e1fd-4c08-8f42-712e116a8290?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/064a02fa-e1fd-4c08-8f42-712e116a8290?feedback_type=other)}&nbsp;&nbsp;[](https://go.dosu.dev/discord-bot)&nbsp;[](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify-official-plugins/issues/2044)

yindo commented 2026-02-16 10:20:33 -05:00

Author

Owner

Copy Link

Copy Source

@davidkhala commented on GitHub (Nov 11, 2025):

For human reading this, can I say the masking is due to Data Source as File? but the Doc Extrator can get raw email when follwing File data source, while dify_extractor cannot

Please confirm my guessing
@dosu

@davidkhala commented on GitHub (Nov 11, 2025): For human reading this, can I say the masking is due to Data Source as File? but the `Doc Extrator` can get raw email when follwing File data source, while dify_extractor cannot Please confirm my guessing @dosu

yindo commented 2026-02-16 10:20:33 -05:00

Author

Owner

Copy Link

Copy Source

@dosubot[bot] commented on GitHub (Nov 11, 2025):

The masking of email addresses isn't caused by using "Data Source as File" or by the dify_extractor itself. Instead, it's controlled by your pipeline's preprocessing configuration—specifically, the 'remove_urls_emails' rule in Dify's CleanProcessor. If you see raw emails with Doc Extractor but masked emails with dify_extractor, it's almost certainly because the pipeline or dataset config for each extractor is different: one has email masking enabled, the other does not. You can adjust this behavior by modifying your pipeline's preprocessing rules, not the extractor plugin itself. For details, see the CleanProcessor logic in Dify's codebase here.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}   

@dosubot[bot] commented on GitHub (Nov 11, 2025): <!-- Answer --> The masking of email addresses isn't caused by using "Data Source as File" or by the dify_extractor itself. Instead, it's controlled by your pipeline's preprocessing configuration—specifically, the 'remove_urls_emails' rule in Dify's CleanProcessor. If you see raw emails with Doc Extractor but masked emails with dify_extractor, it's almost certainly because the pipeline or dataset config for each extractor is different: one has email masking enabled, the other does not. You can adjust this behavior by modifying your pipeline's preprocessing rules, not the extractor plugin itself. For details, see the CleanProcessor logic in Dify's codebase [here](https://github.com/langgenius/dify/blob/5208867ccc7419402bf48afb7d273acf805fc75b/api/core/rag/cleaner/clean_processor.py#L16-L50). <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- ^{How did I do? [Good](https://app.dosu.dev/response-feedback/7263c8e6-5a5c-4fa0-8c2f-76bb698a35a4?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/7263c8e6-5a5c-4fa0-8c2f-76bb698a35a4?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/7263c8e6-5a5c-4fa0-8c2f-76bb698a35a4?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/7263c8e6-5a5c-4fa0-8c2f-76bb698a35a4?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/7263c8e6-5a5c-4fa0-8c2f-76bb698a35a4?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/7263c8e6-5a5c-4fa0-8c2f-76bb698a35a4?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/7263c8e6-5a5c-4fa0-8c2f-76bb698a35a4?feedback_type=other)}&nbsp;&nbsp;[](https://go.dosu.dev/discord-bot)&nbsp;[](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify-official-plugins/issues/2044)

yindo commented 2026-02-16 10:20:33 -05:00

Author

Owner

Copy Link

Copy Source

@davidkhala commented on GitHub (Nov 11, 2025):

@dosu I didn't call CleanProcessor as a step. If so, where can I configure it?

@davidkhala commented on GitHub (Nov 11, 2025): @dosu I didn't call CleanProcessor as a step. If so, where can I configure it?

yindo commented 2026-02-16 10:20:33 -05:00

Author

Owner

Copy Link

Copy Source

@dosubot[bot] commented on GitHub (Nov 11, 2025):

You don't need to call CleanProcessor directly—it's automatically applied as part of Dify's document processing pipeline. You control its behavior (including email masking) via the pipeline or dataset configuration, specifically in the process_rule section. To disable email masking, set the remove_urls_emails rule's enabled field to false in your configuration. For example:

"process_rule": {
    "mode": "custom",
    "rules": {
        "pre_processing_rules": [
            {
                "id": "remove_extra_spaces",
                "enabled": true
            },
            {
                "id": "remove_urls_emails",
                "enabled": false
            }
        ],
        "segmentation": {
            "separator": "\n",
            "max_tokens": 1000
        }
    }
}

This config is usually found in your dataset or pipeline YAML/JSON, or as environment variables in Docker Compose. After editing, restart your Dify services to apply the change. More details and examples are available in this discussion: Dify issue #11861.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}   

@dosubot[bot] commented on GitHub (Nov 11, 2025): <!-- Answer --> You don't need to call CleanProcessor directly—it's automatically applied as part of Dify's document processing pipeline. You control its behavior (including email masking) via the pipeline or dataset configuration, specifically in the process_rule section. To disable email masking, set the remove_urls_emails rule's enabled field to false in your configuration. For example: ```json "process_rule": { "mode": "custom", "rules": { "pre_processing_rules": [ { "id": "remove_extra_spaces", "enabled": true }, { "id": "remove_urls_emails", "enabled": false } ], "segmentation": { "separator": "\n", "max_tokens": 1000 } } } ``` This config is usually found in your dataset or pipeline YAML/JSON, or as environment variables in Docker Compose. After editing, restart your Dify services to apply the change. More details and examples are available in this discussion: [Dify issue #11861](https://github.com/langgenius/dify/issues/11861). <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- ^{How did I do? [Good](https://app.dosu.dev/response-feedback/472c3a2c-df8b-4df1-a885-0ca119e9f4e5?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/472c3a2c-df8b-4df1-a885-0ca119e9f4e5?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/472c3a2c-df8b-4df1-a885-0ca119e9f4e5?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/472c3a2c-df8b-4df1-a885-0ca119e9f4e5?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/472c3a2c-df8b-4df1-a885-0ca119e9f4e5?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/472c3a2c-df8b-4df1-a885-0ca119e9f4e5?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/472c3a2c-df8b-4df1-a885-0ca119e9f4e5?feedback_type=other)}&nbsp;&nbsp;[](https://go.dosu.dev/discord-bot)&nbsp;[](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify-official-plugins/issues/2044)

yindo commented 2026-02-16 10:20:33 -05:00

Author

Owner

Copy Link

Copy Source

@davidkhala commented on GitHub (Nov 11, 2025):

@dosu no such section in exported pipeline DSL.

@davidkhala commented on GitHub (Nov 11, 2025): @dosu no such section in exported pipeline DSL.

yindo commented 2026-02-16 10:20:33 -05:00

Author

Owner

Copy Link

Copy Source

@dosubot[bot] commented on GitHub (Nov 11, 2025):

The exported pipeline DSL often doesn't include a process_rule or pre-processing rules section because Dify manages these rules separately—typically at the dataset level or within workflow nodes, not as part of the pipeline DSL export. Email masking (remove_urls_emails) is controlled by these dataset process rules, which you can configure in the dataset creation or editing UI. If you don't see this section in your DSL, it's likely because the export process doesn't serialize these settings—they're stored in the database and applied dynamically during document processing, not in the exported YAML/JSON itself. For file-based knowledge bases, you can set pre-processing rules (like disabling email masking) before uploading, but for some sources (like Notion), the UI may not expose these options, which is a known limitation or UI bug in certain versions (source).

If you want to ensure email addresses are not masked, adjust the pre-processing rules in your dataset settings before uploading files. If you need to control this via API, include a process_rule section in your upload payload (example). The absence of process_rule in the exported DSL is expected behavior and not a bug.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}   

@dosubot[bot] commented on GitHub (Nov 11, 2025): <!-- Answer --> The exported pipeline DSL often doesn't include a process_rule or pre-processing rules section because Dify manages these rules separately—typically at the dataset level or within workflow nodes, not as part of the pipeline DSL export. Email masking (remove_urls_emails) is controlled by these dataset process rules, which you can configure in the dataset creation or editing UI. If you don't see this section in your DSL, it's likely because the export process doesn't serialize these settings—they're stored in the database and applied dynamically during document processing, not in the exported YAML/JSON itself. For file-based knowledge bases, you can set pre-processing rules (like disabling email masking) before uploading, but for some sources (like Notion), the UI may not expose these options, which is a known limitation or UI bug in certain versions ([source](https://github.com/langgenius/dify/issues/15949#issuecomment-2727609040)). If you want to ensure email addresses are not masked, adjust the pre-processing rules in your dataset settings before uploading files. If you need to control this via API, include a process_rule section in your upload payload ([example](https://github.com/langgenius/dify/discussions/26265)). The absence of process_rule in the exported DSL is expected behavior and not a bug. <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- ^{How did I do? [Good](https://app.dosu.dev/response-feedback/fb4c7f28-ac36-41c2-b928-bfa7d44600ef?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/fb4c7f28-ac36-41c2-b928-bfa7d44600ef?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/fb4c7f28-ac36-41c2-b928-bfa7d44600ef?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/fb4c7f28-ac36-41c2-b928-bfa7d44600ef?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/fb4c7f28-ac36-41c2-b928-bfa7d44600ef?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/fb4c7f28-ac36-41c2-b928-bfa7d44600ef?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/fb4c7f28-ac36-41c2-b928-bfa7d44600ef?feedback_type=other)}&nbsp;&nbsp;[](https://go.dosu.dev/discord-bot)&nbsp;[](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify-official-plugins/issues/2044)

yindo referenced this issue 2026-02-16 10:23:18 -05:00

[PR #799] [MERGED] feat: add plugin for Jira #1556

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

dependabot/uv/datasources/google_cloud_storage/uv-0225d4cafc

dependabot/uv/tools/chart/uv-cdf8d38825

feat/moonshot-kimi-k3-k27-code

feat/gemini-system-instruction-semantics

fix/gemini-text-system-parts

dosu/doc-update-3436

feat/openai-gpt-5-6-support

fix/outlook-trigger-readme-permissions

fix/microsoft-oauth-datasource-tenant

fix/byteplus-api-base-url-label

feat/ark-video-generation-custom-models

fix/openai-stream-reasoning-thinking

fix/skip-live-tests-missing-keys

codex/enhance-discord-integrations

codex/harden-trigger-oauth-permissions

update-lemonade-default-port

codex/fix-pr-review-ci

feat/ark-multimodal-polling

copilot/update-mineru-3x-plugin-file-types

feat/action-security-fix

feat/claude-fable5

codex/fix-ollama-model-category

codex/upgrade-ollama-latest

feat/minimax-m3

codex/refine-plugin-readmes

fix/fireworks-model-label-zh-hans

chore/version-bump-plugins-batch-06

chore/version-bump-plugins-batch-05

chore/version-bump-plugins-batch-04

chore/version-bump-plugins-batch-03

chore/version-bump-plugins-batch-02

fix/plugin-sdk-i18n-fields

chore/version-bump-plugins-batch-01

chore/dependencies-high-priority

chore/dependencies-medium-priority

chore/dependencies-low-priority-clean

chore/dependencies-low-priority

fix/tongyi-align-deepseek-model-metadata

codex/update-openrouter-models

codex/azure-ai-studio-cjk-version-bump

feat/ernie-image-tool

claude/cranky-thompson-b6a281

fix/remove-unavailable-volcengine-models

fix/legacy-plugin-daemon-compatibility

copilot/add-deepseek-v4-to-openrouter

codex/merge-tongyi-tts-prs

codex/depot-runners

chore/migrate-selected-plugins-to-uv-lock

fix/repair-failed-plugin-deploys

ci/fix-plugin-change-detection

chore/migrate-aws-s3-storage-to-uv-lock

chore/migrate-somark-dependencies-to-uv-lock

ci/uv-lock-dependency-validation

fix/2911-openai-api-compatible-tool-call-content

chore/remove-deprecated-plugins

copilot/add-more-models-openrouter-provider

copilot/fix-error-connecting-fireworks-ai

fix/openai-compatible-max-completion-tokens-validation

fix/gemini-lockfile

chore/add-React-fallback-desc

feat/add-gemini-embedding-2-preview

copilot/add-claude-sonnet-opus-4-6

codex/update-xai-models-2721

fix/remove-sentence-transformer-for-hubspot

feat/add-gpt-5.4

copilot/fix-ci-errors-and-bump-version

copilot/fix-security-errors-combined

copilot/update-gemini-model-version

fix/anthropic-parallel-tool-call

codex/fix-azure-ai-studio-azure-core-1-38

codex/fix-azure-blob-azure-core-1-38

codex/fix-azure-openai-azure-core-1-38

gitlab-credentials-c

fix/openrouter-validation-model

fix/gemini-null-checks

fix/deepseek-tool-message-ignore

fix/gemini-dependencies

No results found.

Labels

Clear labels

bug

bug

bug

dependencies

documentation

enhancement

good first issue

invalid

pull-request

Mirrored from GitHub Pull Request

question

stale

No Label enhancement

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

yindo

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: langgenius/dify-official-plugins#799