There is a problem with URLs being removed from responses during segmentation #7302

Closed
opened 2026-02-21 18:19:54 -05:00 by yindo · 6 comments
Owner

Originally created by @doboncho on GitHub (Dec 19, 2024).

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue,否则会被关闭。谢谢!:)
  • Please do not modify this template :) and fill in all the required fields.

Dify version

"server_version": "0.11.0"

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

There is a problem with URLs being removed from responses during segmentation, what is the recommended format for retaining URLs, is Markdown formatting or special character escaping required?

✔️ Expected Behavior

URLs remain in the responses even after segmentation.

Actual Behavior

No response

Originally created by @doboncho on GitHub (Dec 19, 2024). ### Self Checks - [X] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [X] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [X] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [X] [FOR CHINESE USERS] 请务必使用英文提交 Issue,否则会被关闭。谢谢!:) - [X] Please do not modify this template :) and fill in all the required fields. ### Dify version "server_version": "0.11.0" ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce There is a problem with URLs being removed from responses during segmentation, what is the recommended format for retaining URLs, is Markdown formatting or special character escaping required? ### ✔️ Expected Behavior URLs remain in the responses even after segmentation. ### ❌ Actual Behavior _No response_
yindo added the 🐞 bug label 2026-02-21 18:19:54 -05:00
yindo closed this issue 2026-02-21 18:19:54 -05:00
Author
Owner

@dosubot[bot] commented on GitHub (Dec 19, 2024):

I found a similar unsolved discussion regarding URLs being removed from responses during segmentation. You can view it here [1].

To address the issue of URLs being removed during segmentation, you can adjust the configuration settings in Dify. Specifically, you can modify the process_rule configuration to retain URLs by setting the remove_urls_emails rule to false. Here is an example configuration:

"process_rule": {
    "mode": "custom",
    "rules": {
        "pre_processing_rules": [
            {
                "id": "remove_extra_spaces",
                "enabled": true
            },
            {
                "id": "remove_urls_emails",
                "enabled": false
            }
        ],
        "segmentation": {
            "separator": "\n",
            "max_tokens": 1000
        }
    }
}

This configuration will prevent URLs from being removed during the segmentation process [2].

To continue talking to Dosu, mention @dosu.


Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other

@dosubot[bot] commented on GitHub (Dec 19, 2024): <!-- Answer --> I found a similar unsolved discussion regarding URLs being removed from responses during segmentation. You can view it [here](https://github.com/langgenius/dify/discussions/11856) <sup>[[1]](https://github.com/langgenius/dify/discussions/11856)</sup>. To address the issue of URLs being removed during segmentation, you can adjust the configuration settings in Dify. Specifically, you can modify the `process_rule` configuration to retain URLs by setting the `remove_urls_emails` rule to `false`. Here is an example configuration: ```json "process_rule": { "mode": "custom", "rules": { "pre_processing_rules": [ { "id": "remove_extra_spaces", "enabled": true }, { "id": "remove_urls_emails", "enabled": false } ], "segmentation": { "separator": "\n", "max_tokens": 1000 } } } ``` This configuration will prevent URLs from being removed during the segmentation process <sup>[[2]](https://github.com/langgenius/dify/blob/main/web/app/(commonLayout)/datasets/template/template.en.mdx)</sup>. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/7ba7074e-b52d-4317-b2a6-5082affc324d?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/7ba7074e-b52d-4317-b2a6-5082affc324d?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/7ba7074e-b52d-4317-b2a6-5082affc324d?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/7ba7074e-b52d-4317-b2a6-5082affc324d?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/7ba7074e-b52d-4317-b2a6-5082affc324d?feedback_type=hallucination) | [Other](https://app.dosu.dev/response-feedback/7ba7074e-b52d-4317-b2a6-5082affc324d?feedback_type=other)</sup>
Author
Owner

@doboncho commented on GitHub (Dec 19, 2024):

How do we change the process_rule configuration? Could you be more specific?

2024年12月20日(金) 9:38 dosubot[bot] @.***>:

I found a similar unsolved discussion regarding URLs being removed from
responses during segmentation. You can view it here
https://github.com/langgenius/dify/discussions/11856 [1]
https://github.com/langgenius/dify/discussions/11856.

To address the issue of URLs being removed during segmentation, you can
adjust the configuration settings in Dify. Specifically, you can modify the
process_rule configuration to retain URLs by setting the
remove_urls_emails rule to false. Here is an example configuration:

"process_rule": {
"mode": "custom",
"rules": {
"pre_processing_rules": [
{
"id": "remove_extra_spaces",
"enabled": true
},
{
"id": "remove_urls_emails",
"enabled": false
}
],
"segmentation": {
"separator": "\n",
"max_tokens": 1000
}
}
}

This configuration will prevent URLs from being removed during the
segmentation process [2]
https://github.com/langgenius/dify/blob/main/web/app/(commonLayout)/datasets/template/template.en.mdx
.

To continue talking to Dosu https://dosu.dev, mention @dosu
https://github.com/dosu.

Help Dosu learn! Give it feedback: Great Response
https://app.dosu.dev/response-feedback/7ba7074e-b52d-4317-b2a6-5082affc324d?feedback_type=great_response
| Irrelevant Answer
https://app.dosu.dev/response-feedback/7ba7074e-b52d-4317-b2a6-5082affc324d?feedback_type=irrelevant_answer
| Incorrect Sources
https://app.dosu.dev/response-feedback/7ba7074e-b52d-4317-b2a6-5082affc324d?feedback_type=incorrect_sources
| Too Verbose
https://app.dosu.dev/response-feedback/7ba7074e-b52d-4317-b2a6-5082affc324d?feedback_type=too_verbose
| Hallucination
https://app.dosu.dev/response-feedback/7ba7074e-b52d-4317-b2a6-5082affc324d?feedback_type=hallucination
| Other
https://app.dosu.dev/response-feedback/7ba7074e-b52d-4317-b2a6-5082affc324d?feedback_type=other


Reply to this email directly, view it on GitHub
https://github.com/langgenius/dify/issues/11861#issuecomment-2556022708,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/APHDDGVDASHPP5Y7VD6BEDL2GNRG5AVCNFSM6AAAAABT6C2VR2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNJWGAZDENZQHA
.
You are receiving this because you authored the thread.Message ID:
@.***>

--
松永 力

@doboncho commented on GitHub (Dec 19, 2024): How do we change the process_rule configuration? Could you be more specific? 2024年12月20日(金) 9:38 dosubot[bot] ***@***.***>: > I found a similar unsolved discussion regarding URLs being removed from > responses during segmentation. You can view it here > <https://github.com/langgenius/dify/discussions/11856> [1] > <https://github.com/langgenius/dify/discussions/11856>. > > To address the issue of URLs being removed during segmentation, you can > adjust the configuration settings in Dify. Specifically, you can modify the > process_rule configuration to retain URLs by setting the > remove_urls_emails rule to false. Here is an example configuration: > > "process_rule": { > "mode": "custom", > "rules": { > "pre_processing_rules": [ > { > "id": "remove_extra_spaces", > "enabled": true > }, > { > "id": "remove_urls_emails", > "enabled": false > } > ], > "segmentation": { > "separator": "\n", > "max_tokens": 1000 > } > } > } > > This configuration will prevent URLs from being removed during the > segmentation process [2] > <https://github.com/langgenius/dify/blob/main/web/app/(commonLayout)/datasets/template/template.en.mdx> > . > > *To continue talking to Dosu <https://dosu.dev>, mention @dosu > <https://github.com/dosu>.* > ------------------------------ > > Help Dosu learn! Give it feedback: Great Response > <https://app.dosu.dev/response-feedback/7ba7074e-b52d-4317-b2a6-5082affc324d?feedback_type=great_response> > | Irrelevant Answer > <https://app.dosu.dev/response-feedback/7ba7074e-b52d-4317-b2a6-5082affc324d?feedback_type=irrelevant_answer> > | Incorrect Sources > <https://app.dosu.dev/response-feedback/7ba7074e-b52d-4317-b2a6-5082affc324d?feedback_type=incorrect_sources> > | Too Verbose > <https://app.dosu.dev/response-feedback/7ba7074e-b52d-4317-b2a6-5082affc324d?feedback_type=too_verbose> > | Hallucination > <https://app.dosu.dev/response-feedback/7ba7074e-b52d-4317-b2a6-5082affc324d?feedback_type=hallucination> > | Other > <https://app.dosu.dev/response-feedback/7ba7074e-b52d-4317-b2a6-5082affc324d?feedback_type=other> > > — > Reply to this email directly, view it on GitHub > <https://github.com/langgenius/dify/issues/11861#issuecomment-2556022708>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/APHDDGVDASHPP5Y7VD6BEDL2GNRG5AVCNFSM6AAAAABT6C2VR2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNJWGAZDENZQHA> > . > You are receiving this because you authored the thread.Message ID: > ***@***.***> > -- 松永 力
Author
Owner

@dosubot[bot] commented on GitHub (Dec 19, 2024):

To change the process_rule configuration in Dify, you need to modify the configuration file where these settings are defined. Here’s a step-by-step guide:

  1. Locate the Configuration File: The process_rule settings are typically part of a configuration file in your Dify setup. If you are using a self-hosted Docker deployment, these settings might be in a JSON or YAML file within your project directory.

  2. Edit the Configuration: Open the configuration file and locate the process_rule section. Modify the remove_urls_emails rule by setting its enabled field to false. Here’s an example configuration:

    {
      "process_rule": {
        "mode": "custom",
        "rules": {
          "pre_processing_rules": [
            {
              "id": "remove_extra_spaces",
              "enabled": true
            },
            {
              "id": "remove_urls_emails",
              "enabled": false
            }
          ],
          "segmentation": {
            "separator": "\n",
            "max_tokens": 1000
          }
        }
      }
    }
    
  3. Apply the Changes: If you are using Docker, you will need to restart your Docker containers to apply the changes. Use the following commands:

    docker compose down
    docker compose up -d
    

This process will ensure that URLs are retained during the segmentation process [1].

To continue talking to Dosu, mention @dosu.


Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other

@dosubot[bot] commented on GitHub (Dec 19, 2024): <!-- Answer --> To change the `process_rule` configuration in Dify, you need to modify the configuration file where these settings are defined. Here’s a step-by-step guide: 1. **Locate the Configuration File**: The `process_rule` settings are typically part of a configuration file in your Dify setup. If you are using a self-hosted Docker deployment, these settings might be in a JSON or YAML file within your project directory. 2. **Edit the Configuration**: Open the configuration file and locate the `process_rule` section. Modify the `remove_urls_emails` rule by setting its `enabled` field to `false`. Here’s an example configuration: ```json { "process_rule": { "mode": "custom", "rules": { "pre_processing_rules": [ { "id": "remove_extra_spaces", "enabled": true }, { "id": "remove_urls_emails", "enabled": false } ], "segmentation": { "separator": "\n", "max_tokens": 1000 } } } } ``` 3. **Apply the Changes**: If you are using Docker, you will need to restart your Docker containers to apply the changes. Use the following commands: ```bash docker compose down docker compose up -d ``` This process will ensure that URLs are retained during the segmentation process <sup>[[1]](https://github.com/langgenius/dify/blob/main/web/app/(commonLayout)/datasets/template/template.en.mdx)</sup>. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/7e030d7b-e560-4cf0-8c89-57fb57db16a2?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/7e030d7b-e560-4cf0-8c89-57fb57db16a2?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/7e030d7b-e560-4cf0-8c89-57fb57db16a2?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/7e030d7b-e560-4cf0-8c89-57fb57db16a2?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/7e030d7b-e560-4cf0-8c89-57fb57db16a2?feedback_type=hallucination) | [Other](https://app.dosu.dev/response-feedback/7e030d7b-e560-4cf0-8c89-57fb57db16a2?feedback_type=other)</sup>
Author
Owner

@doboncho commented on GitHub (Dec 19, 2024):

The process_rule section seems to be in json format. If I were to write it in docker-compose.yaml, how would it look like?

@doboncho commented on GitHub (Dec 19, 2024): The process_rule section seems to be in json format. If I were to write it in docker-compose.yaml, how would it look like?
Author
Owner

@doboncho commented on GitHub (Dec 19, 2024):

The process_rule section seems to be in json format. If I were to write it
in docker-compose.yaml, how would it look like?

2024年12月20日(金) 9:52 dosubot[bot] @.***>:

To change the process_rule configuration in Dify, you need to modify the
configuration file where these settings are defined. Here’s a step-by-step
guide:

Locate the Configuration File: The process_rule settings are
typically part of a configuration file in your Dify setup. If you are using
a self-hosted Docker deployment, these settings might be in a JSON or YAML
file within your project directory.
2.

Edit the Configuration: Open the configuration file and locate the
process_rule section. Modify the remove_urls_emails rule by setting
its enabled field to false. Here’s an example configuration:

{
"process_rule": {
"mode": "custom",
"rules": {
"pre_processing_rules": [
{
"id": "remove_extra_spaces",
"enabled": true
},
{
"id": "remove_urls_emails",
"enabled": false
}
],
"segmentation": {
"separator": "\n",
"max_tokens": 1000
}
}
}
}

Apply the Changes: If you are using Docker, you will need to restart
your Docker containers to apply the changes. Use the following commands:

docker compose down
docker compose up -d

This process will ensure that URLs are retained during the segmentation
process [1]
https://github.com/langgenius/dify/blob/main/web/app/(commonLayout)/datasets/template/template.en.mdx
.

To continue talking to Dosu https://dosu.dev, mention @dosu
https://github.com/dosu.

Help Dosu learn! Give it feedback: Great Response
https://app.dosu.dev/response-feedback/7e030d7b-e560-4cf0-8c89-57fb57db16a2?feedback_type=great_response
| Irrelevant Answer
https://app.dosu.dev/response-feedback/7e030d7b-e560-4cf0-8c89-57fb57db16a2?feedback_type=irrelevant_answer
| Incorrect Sources
https://app.dosu.dev/response-feedback/7e030d7b-e560-4cf0-8c89-57fb57db16a2?feedback_type=incorrect_sources
| Too Verbose
https://app.dosu.dev/response-feedback/7e030d7b-e560-4cf0-8c89-57fb57db16a2?feedback_type=too_verbose
| Hallucination
https://app.dosu.dev/response-feedback/7e030d7b-e560-4cf0-8c89-57fb57db16a2?feedback_type=hallucination
| Other
https://app.dosu.dev/response-feedback/7e030d7b-e560-4cf0-8c89-57fb57db16a2?feedback_type=other


Reply to this email directly, view it on GitHub
https://github.com/langgenius/dify/issues/11861#issuecomment-2556036593,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/APHDDGQ5FHWQRTSOSKXIFZD2GNS5XAVCNFSM6AAAAABT6C2VR2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNJWGAZTMNJZGM
.
You are receiving this because you authored the thread.Message ID:
@.***>

--
松永 力

@doboncho commented on GitHub (Dec 19, 2024): The process_rule section seems to be in json format. If I were to write it in docker-compose.yaml, how would it look like? 2024年12月20日(金) 9:52 dosubot[bot] ***@***.***>: > To change the process_rule configuration in Dify, you need to modify the > configuration file where these settings are defined. Here’s a step-by-step > guide: > > 1. > > *Locate the Configuration File*: The process_rule settings are > typically part of a configuration file in your Dify setup. If you are using > a self-hosted Docker deployment, these settings might be in a JSON or YAML > file within your project directory. > 2. > > *Edit the Configuration*: Open the configuration file and locate the > process_rule section. Modify the remove_urls_emails rule by setting > its enabled field to false. Here’s an example configuration: > > { > "process_rule": { > "mode": "custom", > "rules": { > "pre_processing_rules": [ > { > "id": "remove_extra_spaces", > "enabled": true > }, > { > "id": "remove_urls_emails", > "enabled": false > } > ], > "segmentation": { > "separator": "\n", > "max_tokens": 1000 > } > } > } > } > > 3. > > *Apply the Changes*: If you are using Docker, you will need to restart > your Docker containers to apply the changes. Use the following commands: > > docker compose down > docker compose up -d > > > This process will ensure that URLs are retained during the segmentation > process [1] > <https://github.com/langgenius/dify/blob/main/web/app/(commonLayout)/datasets/template/template.en.mdx> > . > > *To continue talking to Dosu <https://dosu.dev>, mention @dosu > <https://github.com/dosu>.* > ------------------------------ > > Help Dosu learn! Give it feedback: Great Response > <https://app.dosu.dev/response-feedback/7e030d7b-e560-4cf0-8c89-57fb57db16a2?feedback_type=great_response> > | Irrelevant Answer > <https://app.dosu.dev/response-feedback/7e030d7b-e560-4cf0-8c89-57fb57db16a2?feedback_type=irrelevant_answer> > | Incorrect Sources > <https://app.dosu.dev/response-feedback/7e030d7b-e560-4cf0-8c89-57fb57db16a2?feedback_type=incorrect_sources> > | Too Verbose > <https://app.dosu.dev/response-feedback/7e030d7b-e560-4cf0-8c89-57fb57db16a2?feedback_type=too_verbose> > | Hallucination > <https://app.dosu.dev/response-feedback/7e030d7b-e560-4cf0-8c89-57fb57db16a2?feedback_type=hallucination> > | Other > <https://app.dosu.dev/response-feedback/7e030d7b-e560-4cf0-8c89-57fb57db16a2?feedback_type=other> > > — > Reply to this email directly, view it on GitHub > <https://github.com/langgenius/dify/issues/11861#issuecomment-2556036593>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/APHDDGQ5FHWQRTSOSKXIFZD2GNS5XAVCNFSM6AAAAABT6C2VR2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNJWGAZTMNJZGM> > . > You are receiving this because you authored the thread.Message ID: > ***@***.***> > -- 松永 力
Author
Owner

@dosubot[bot] commented on GitHub (Dec 19, 2024):

To convert the process_rule JSON configuration into a docker-compose.yaml format, you can define it as environment variables within the Docker Compose file. Here's how you can structure it:

version: '3.8'
services:
  your_service_name:
    image: your_image_name
    environment:
      PROCESS_RULE_MODE: "custom"
      PRE_PROCESSING_RULES: |
        - id: remove_extra_spaces
          enabled: true
        - id: remove_urls_emails
          enabled: false
      SEGMENTATION_SEPARATOR: "\n"
      SEGMENTATION_MAX_TOKENS: 1000

Replace your_service_name and your_image_name with the appropriate values for your Docker setup. This YAML format sets environment variables that correspond to the JSON structure you provided. After making these changes, remember to restart your Docker containers using docker compose down followed by docker compose up -d to apply the new configuration.

To continue talking to Dosu, mention @dosu.


Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other

@dosubot[bot] commented on GitHub (Dec 19, 2024): <!-- Answer --> To convert the `process_rule` JSON configuration into a `docker-compose.yaml` format, you can define it as environment variables within the Docker Compose file. Here's how you can structure it: ```yaml version: '3.8' services: your_service_name: image: your_image_name environment: PROCESS_RULE_MODE: "custom" PRE_PROCESSING_RULES: | - id: remove_extra_spaces enabled: true - id: remove_urls_emails enabled: false SEGMENTATION_SEPARATOR: "\n" SEGMENTATION_MAX_TOKENS: 1000 ``` Replace `your_service_name` and `your_image_name` with the appropriate values for your Docker setup. This YAML format sets environment variables that correspond to the JSON structure you provided. After making these changes, remember to restart your Docker containers using `docker compose down` followed by `docker compose up -d` to apply the new configuration. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/99571bd8-9ca3-4390-abb5-3224ac2e48c1?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/99571bd8-9ca3-4390-abb5-3224ac2e48c1?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/99571bd8-9ca3-4390-abb5-3224ac2e48c1?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/99571bd8-9ca3-4390-abb5-3224ac2e48c1?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/99571bd8-9ca3-4390-abb5-3224ac2e48c1?feedback_type=hallucination) | [Other](https://app.dosu.dev/response-feedback/99571bd8-9ca3-4390-abb5-3224ac2e48c1?feedback_type=other)</sup>
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#7302