langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-22 20:45:26 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

"{" Character Removed When Adding Dataset via API #7304

New Issue
Closed
opened 2026-02-21 18:19:55 -05:00 by yindo · 4 comments
yindo commented 2026-02-21 18:19:55 -05:00
Owner
Copy Link
Copy Source

Originally created by @pickuse2013 on GitHub (Dec 19, 2024).

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

0.14.1

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

When attempting to update documents through the Knowledge API, the "{" character in the input text is incorrectly removed.

This issue occurred after updating from version v0.13.2 to v0.14.1.

Affected API:
/datasets/{dataset_id}/documents/{document_id}/update-by-text

Steps to Reproduce:

  1. Use the following input data to call the API:
{
    "name": "test",
    "text": "{aaaa}",
    "indexing_technique": "high_quality",
    "process_rule": {
        "mode": "custom",
        "rules": {
            "pre_processing_rules": [
                {
                    "id": "remove_extra_spaces",
                    "enabled": true
                },
                {
                    "id": "remove_urls_emails",
                    "enabled": false
                }
            ],
            "segmentation": {
                "separator": "\n",
                "max_tokens": 10000,
                "chunk_overlap": 100
            }
        }
    }
}
  1. Check the stored document in the dataset.

✔️ Expected Behavior

The text in the document should remain as {aaaa}.

❌ Actual Behavior

The stored text in the document becomes aaaa}.

Originally created by @pickuse2013 on GitHub (Dec 19, 2024). ### Self Checks - [X] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [X] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [X] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [X] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:） - [X] Please do not modify this template :) and fill in all the required fields. ### Dify version 0.14.1 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce When attempting to update documents through the Knowledge API, the "{" character in the input text is incorrectly removed. This issue occurred after updating from version `v0.13.2` to `v0.14.1`. **Affected API:** `/datasets/{dataset_id}/documents/{document_id}/update-by-text` **Steps to Reproduce:** 1. Use the following input data to call the API: ```json { "name": "test", "text": "{aaaa}", "indexing_technique": "high_quality", "process_rule": { "mode": "custom", "rules": { "pre_processing_rules": [ { "id": "remove_extra_spaces", "enabled": true }, { "id": "remove_urls_emails", "enabled": false } ], "segmentation": { "separator": "\n", "max_tokens": 10000, "chunk_overlap": 100 } } } } ``` 2. Check the stored document in the dataset. ### ✔️ Expected Behavior The text in the document should remain as `{aaaa}`. ### ❌ Actual Behavior The stored text in the document becomes `aaaa}`.
yindo added the 🐞 bug label 2026-02-21 18:19:55 -05:00
yindo closed this issue 2026-02-21 18:19:55 -05:00
yindo commented 2026-02-21 18:19:55 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Dec 19, 2024):

@pickuse2013 Hello there! I'm here to help you with bugs, answer your questions, and even guide you on how to become a contributor. Let's figure out the issue you're facing together.

The issue you're experiencing with the "{" character being removed when updating documents via the Knowledge API in version 0.14.1 might be related to changes in the text processing pipeline between versions v0.13.2 and v0.14.1. However, I couldn't find specific information about changes affecting character handling, particularly for curly braces {}.

To address this, you might want to check the process_rule configuration to ensure no unintended pre-processing rules are affecting the text. Since the remove_extra_spaces rule is enabled, it primarily targets spaces and newlines, but it shouldn't affect curly braces. The remove_urls_emails rule is disabled, so it shouldn't impact the text either.

If the issue persists, consider reviewing the text processing logic in the Dify codebase, particularly around the update-by-text endpoint, to identify any unintended transformations or bugs introduced in the latest version.

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other

@dosubot[bot] commented on GitHub (Dec 19, 2024): <!-- Greeting --> @pickuse2013 Hello there! I'm here to help you with bugs, answer your questions, and even guide you on how to become a contributor. Let's figure out the issue you're facing together. <!-- Answer --> The issue you're experiencing with the "{" character being removed when updating documents via the Knowledge API in version 0.14.1 might be related to changes in the text processing pipeline between versions v0.13.2 and v0.14.1. However, I couldn't find specific information about changes affecting character handling, particularly for curly braces `{}`. To address this, you might want to check the `process_rule` configuration to ensure no unintended pre-processing rules are affecting the text. Since the `remove_extra_spaces` rule is enabled, it primarily targets spaces and newlines, but it shouldn't affect curly braces. The `remove_urls_emails` rule is disabled, so it shouldn't impact the text either. If the issue persists, consider reviewing the text processing logic in the Dify codebase, particularly around the `update-by-text` endpoint, to identify any unintended transformations or bugs introduced in the latest version. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/50d62f1e-e764-42c8-b91b-8dce862d3c5a?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/50d62f1e-e764-42c8-b91b-8dce862d3c5a?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/50d62f1e-e764-42c8-b91b-8dce862d3c5a?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/50d62f1e-e764-42c8-b91b-8dce862d3c5a?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/50d62f1e-e764-42c8-b91b-8dce862d3c5a?feedback_type=hallucination) | [Other](https://app.dosu.dev/response-feedback/50d62f1e-e764-42c8-b91b-8dce862d3c5a?feedback_type=other)</sup>
yindo commented 2026-02-21 18:19:55 -05:00
Author
Owner
Copy Link
Copy Source

@Augus commented on GitHub (Dec 24, 2024):

@takatost or anyone, could you take a look at this issue? This is a critical problem—updating the document leads to a loss of content integrity. The issue occurs when upgrading from v0.13 to v0.14.1. Based on the latest release notes, it seems that it hasn't been fixed in v0.14.2 either.

@Augus commented on GitHub (Dec 24, 2024): @takatost or anyone, could you take a look at this issue? This is a critical problem—updating the document leads to a loss of content integrity. The issue occurs when upgrading from v0.13 to v0.14.1. Based on the latest release notes, it seems that it hasn't been fixed in v0.14.2 either.
yindo commented 2026-02-21 18:19:55 -05:00
Author
Owner
Copy Link
Copy Source

@yihong0618 commented on GitHub (Dec 25, 2024):

will take a look maybe today

@yihong0618 commented on GitHub (Dec 25, 2024): will take a look maybe today
yindo commented 2026-02-21 18:19:55 -05:00
Author
Owner
Copy Link
Copy Source

@zandko commented on GitHub (Dec 25, 2024):

@takatost or anyone, could you take a look at this issue? This is a critical problem—updating the document leads to a loss of content integrity. The issue occurs when upgrading from v0.13 to v0.14.1. Based on the latest release notes, it seems that it hasn't been fixed in v0.14.2 either.
At first, it was to reduce the possibility of such situations occurring in the titles, which would affect the retrieval quality of large models.

@zandko commented on GitHub (Dec 25, 2024): > @takatost or anyone, could you take a look at this issue? This is a critical problem—updating the document leads to a loss of content integrity. The issue occurs when upgrading from v0.13 to v0.14.1. Based on the latest release notes, it seems that it hasn't been fixed in v0.14.2 either. At first, it was to reduce the possibility of such situations occurring in the titles, which would affect the retrieval quality of large models.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#7304
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?