langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-22 12:36:14 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Error in "Text Preprocessing and Cleaning" stage when adding new files to existing knowledge base #5155

New Issue
Closed
opened 2026-02-21 18:09:39 -05:00 by yindo · 5 comments
yindo commented 2026-02-21 18:09:39 -05:00
Owner
Copy Link
Copy Source

Originally created by @cht-k on GitHub (Aug 21, 2024).

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

0.7.1

Cloud or Self Hosted

Cloud

Steps to reproduce

When attempting to add additional documents to an existing Knowledge in Dify Cloud, an error occurs during the "Text Preprocessing and Cleaning" stage. The error message suggests that there's an issue with NLTK (Natural Language Toolkit) being unable to locate a resource named "punkt_tab".

Steps to Reproduce:

  1. Navigate to an existing Knowledge base
  2. Click the "Add file" button
  3. On the "Upload file" page, upload a new document and click "Next"
  4. Error occurs upon entering the "Text Preprocessing and Cleaning" page

image

{
    "code": "indexing_estimate_error",
    "message": "\n**********************************************************************\n  Resource \u001b[93mpunkt_tab\u001b[0m not found.\n  Please use the NLTK Downloader to obtain the resource:\n\n  \u001b[31m>>> import nltk\n  >>> nltk.download('punkt_tab')\n  \u001b[0m\n  For more information see: https://www.nltk.org/data.html\n\n  Attempted to load \u001b[93mtokenizers/punkt_tab/english/\u001b[0m\n\n  Searched in:\n    - '/root/nltk_data'\n    - '/app/api/.venv/nltk_data'\n    - '/app/api/.venv/share/nltk_data'\n    - '/app/api/.venv/lib/nltk_data'\n    - '/usr/share/nltk_data'\n    - '/usr/local/share/nltk_data'\n    - '/usr/lib/nltk_data'\n    - '/usr/local/lib/nltk_data'\n**********************************************************************\n",
    "status": 500
}

✔️ Expected Behavior

Ability to embed new documents and add them to an existing Knowledge base.

❌ Actual Behavior

Error message appears when entering the "Text Preprocessing and Cleaning" page.

Originally created by @cht-k on GitHub (Aug 21, 2024). ### Self Checks - [X] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [X] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [X] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [X] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:） - [X] Please do not modify this template :) and fill in all the required fields. ### Dify version 0.7.1 ### Cloud or Self Hosted Cloud ### Steps to reproduce When attempting to add additional documents to an existing Knowledge in Dify Cloud, an error occurs during the "Text Preprocessing and Cleaning" stage. The error message suggests that there's an issue with NLTK (Natural Language Toolkit) being unable to locate a resource named "punkt_tab". Steps to Reproduce: 1. Navigate to an existing Knowledge base 2. Click the "Add file" button 3. On the "Upload file" page, upload a new document and click "Next" 4. Error occurs upon entering the "Text Preprocessing and Cleaning" page ![image](https://github.com/user-attachments/assets/2f2049d1-bb4a-40fc-bb87-1763c9ff02d3) ```json { "code": "indexing_estimate_error", "message": "\n**********************************************************************\n Resource \u001b[93mpunkt_tab\u001b[0m not found.\n Please use the NLTK Downloader to obtain the resource:\n\n \u001b[31m>>> import nltk\n >>> nltk.download('punkt_tab')\n \u001b[0m\n For more information see: https://www.nltk.org/data.html\n\n Attempted to load \u001b[93mtokenizers/punkt_tab/english/\u001b[0m\n\n Searched in:\n - '/root/nltk_data'\n - '/app/api/.venv/nltk_data'\n - '/app/api/.venv/share/nltk_data'\n - '/app/api/.venv/lib/nltk_data'\n - '/usr/share/nltk_data'\n - '/usr/local/share/nltk_data'\n - '/usr/lib/nltk_data'\n - '/usr/local/lib/nltk_data'\n**********************************************************************\n", "status": 500 } ``` ### ✔️ Expected Behavior Ability to embed new documents and add them to an existing Knowledge base. ### ❌ Actual Behavior Error message appears when entering the "Text Preprocessing and Cleaning" page.
yindo added the 🐞 bug label 2026-02-21 18:09:39 -05:00
yindo closed this issue 2026-02-21 18:09:39 -05:00
yindo commented 2026-02-21 18:09:39 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Aug 21, 2024):

Hello @cht-k! I'm here to help you with any bugs, questions, or contributions. Let's work together to solve the issue you're facing.

The error you're encountering indicates that the NLTK resource punkt_tab is missing and needs to be downloaded. The error message provides instructions on how to download the resource using the NLTK Downloader.

To resolve this issue, you can add the following code snippet to your initialization script or setup file to ensure that the punkt_tab resource is downloaded:

import nltk

nltk.download('punkt')

This will download the necessary punkt tokenizer data, which includes punkt_tab.

If you have access to the codebase, you can add this snippet to the appropriate initialization section. If not, you may need to contact the maintainers or support team to have this change implemented.

Currently, there is no specific information in the Dify codebase regarding the handling or downloading of NLTK resources, including punkt_tab [1][2].

To continue talking to Dosu, mention @dosu.

@dosubot[bot] commented on GitHub (Aug 21, 2024): <!-- Greeting --> Hello @cht-k! I'm here to help you with any bugs, questions, or contributions. Let's work together to solve the issue you're facing. <!-- Answer --> The error you're encountering indicates that the NLTK resource `punkt_tab` is missing and needs to be downloaded. The error message provides instructions on how to download the resource using the NLTK Downloader. To resolve this issue, you can add the following code snippet to your initialization script or setup file to ensure that the `punkt_tab` resource is downloaded: ```python import nltk nltk.download('punkt') ``` This will download the necessary `punkt` tokenizer data, which includes `punkt_tab`. If you have access to the codebase, you can add this snippet to the appropriate initialization section. If not, you may need to contact the maintainers or support team to have this change implemented. Currently, there is no specific information in the Dify codebase regarding the handling or downloading of NLTK resources, including `punkt_tab` <sup>[[1]](https://github.com/langgenius/dify/blob/main/api/core/indexing_runner.py)</sup><sup>[[2]](https://github.com/langgenius/dify/blob/main/api/core/rag/cleaner/clean_processor.py)</sup>. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
yindo commented 2026-02-21 18:09:39 -05:00
Author
Owner
Copy Link
Copy Source

@crazywoola commented on GitHub (Aug 21, 2024):

What system processing LLM did you choose?

@crazywoola commented on GitHub (Aug 21, 2024): What system processing LLM did you choose?
yindo commented 2026-02-21 18:09:39 -05:00
Author
Owner
Copy Link
Copy Source

@cht-k commented on GitHub (Aug 22, 2024):

I'm not entirely sure what specific information you're requesting here, but I can provide some details that might be relevant:

  • The embedding model I'm using is OpenAI's text-embedding-3-large.

From my observation, this issue mainly affects the process when the chunk settings are set to "automatic".

It's worth noting that I'm using the Dify Cloud version, and these error messages appear to be generated by Dify Cloud's backend. Therefore, I don't believe my local system information is relevant to this issue.

Could you please clarify if you need any other specific information about the LLM or system settings? I'd be happy to provide more details if I can locate them in the Dify Cloud interface.

@cht-k commented on GitHub (Aug 22, 2024): I'm not entirely sure what specific information you're requesting here, but I can provide some details that might be relevant: - The embedding model I'm using is OpenAI's `text-embedding-3-large`. From my observation, this issue mainly affects the process when the chunk settings are set to "automatic". It's worth noting that I'm using the Dify Cloud version, and these error messages appear to be generated by Dify Cloud's backend. Therefore, I don't believe my local system information is relevant to this issue. Could you please clarify if you need any other specific information about the LLM or system settings? I'd be happy to provide more details if I can locate them in the Dify Cloud interface.
yindo commented 2026-02-21 18:09:40 -05:00
Author
Owner
Copy Link
Copy Source

@tamanobi commented on GitHub (Aug 22, 2024):

I am also facing the same error. I use Dify Cloud. When I create an empty knowledge database, Dify always happens the error.

I was able to avoid the punk_tab error by following these steps:

  • Create a non-empty knowledge database
  • Set chunk settings to custom

If you use HTTP API to create document, choose "custom" mode of process_rule.

@tamanobi commented on GitHub (Aug 22, 2024): I am also facing the same error. I use Dify Cloud. When I create an empty knowledge database, Dify always happens the error. I was able to avoid the punk_tab error by following these steps: * Create a non-empty knowledge database * Set chunk settings to custom If you use HTTP API to create document, choose "custom" mode of process_rule.
yindo commented 2026-02-21 18:09:40 -05:00
Author
Owner
Copy Link
Copy Source

@crazywoola commented on GitHub (Sep 6, 2024):

This should be resolved in #7582

@crazywoola commented on GitHub (Sep 6, 2024): This should be resolved in #7582
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#5155
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?