langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 03:55:25 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

csv chunks turns gibberish #8391

New Issue
Closed
opened 2026-02-21 18:25:26 -05:00 by yindo · 2 comments
yindo commented 2026-02-21 18:25:26 -05:00
Owner
Copy Link
Copy Source

Originally created by @lema-founders on GitHub (Feb 18, 2025).

Originally assigned to: @JohnJyong on GitHub.

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

0.15.3

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

Create a knowledge base and index attached csv file using parent - child chunk strategy (Default settings), any embeddings, and reranker model will do

Audience-1.csv

✔️ Expected Behavior

csv data properly read, creating parent and chunks based on structure (i.e. Women 18 - 24 5, Men 18 - 24 3.1)
Image

❌ Actual Behavior

Chunks become unnamed and nans

Image

Originally created by @lema-founders on GitHub (Feb 18, 2025). Originally assigned to: @JohnJyong on GitHub. ### Self Checks - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:） - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version 0.15.3 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce Create a knowledge base and index attached csv file using parent - child chunk strategy (Default settings), any embeddings, and reranker model will do [Audience-1.csv](https://github.com/user-attachments/files/18848163/Audience-1.csv) ### ✔️ Expected Behavior csv data properly read, creating parent and chunks based on structure (i.e. Women 18 - 24 5, Men 18 - 24 3.1) ![Image](https://github.com/user-attachments/assets/e5d85b15-85b7-41d8-ba93-e7223e51afc0) ### ❌ Actual Behavior Chunks become unnamed and nans ![Image](https://github.com/user-attachments/assets/2267eafc-02e2-41ce-a654-c0ea8ff05303)
yindo added the 🐞 bug label 2026-02-21 18:25:26 -05:00
yindo closed this issue 2026-02-21 18:25:26 -05:00
yindo commented 2026-02-21 18:25:27 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Feb 18, 2025):

The issue you're experiencing with CSV chunks turning into "Unnamed" and "nan" values could be related to how Dify handles CSV file encoding and malformed data. Dify uses UTF-8 encoding with the "ignore" option for errors when processing CSV files. This means that any encoding issues in the CSV file might be ignored, potentially leading to data being misinterpreted or lost [1].

Here are a few steps you can take to troubleshoot and potentially resolve the issue:

  1. Check CSV Encoding: Ensure that your CSV file is correctly encoded in UTF-8. You can use a text editor or a tool like iconv to verify and convert the encoding if necessary.

  2. Inspect CSV Structure: Open the CSV file in a spreadsheet application to check for any structural issues, such as missing headers or irregular row lengths, which might cause parsing errors.

  3. Data Cleaning: Before indexing, clean the CSV data to remove any unnecessary columns or rows that might be causing issues. Ensure that all columns have headers and that there are no empty rows.

  4. Review Chunking Settings: Double-check the chunking settings in Dify to ensure they align with the structure of your CSV file. The default settings for the parent-child chunk strategy involve chunking by paragraphs and sentences, which might not be suitable for all CSV structures [2].

By addressing these areas, you may be able to resolve the issue with the CSV data turning into "Unnamed" and "nan" values during the indexing process.

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other

@dosubot[bot] commented on GitHub (Feb 18, 2025): <!-- Answer --> The issue you're experiencing with CSV chunks turning into "Unnamed" and "nan" values could be related to how Dify handles CSV file encoding and malformed data. Dify uses UTF-8 encoding with the "ignore" option for errors when processing CSV files. This means that any encoding issues in the CSV file might be ignored, potentially leading to data being misinterpreted or lost <sup>[[1]](https://github.com/langgenius/dify/blob/main/api/core/workflow/nodes/document_extractor/node.py)</sup>. Here are a few steps you can take to troubleshoot and potentially resolve the issue: 1. **Check CSV Encoding**: Ensure that your CSV file is correctly encoded in UTF-8. You can use a text editor or a tool like `iconv` to verify and convert the encoding if necessary. 2. **Inspect CSV Structure**: Open the CSV file in a spreadsheet application to check for any structural issues, such as missing headers or irregular row lengths, which might cause parsing errors. 3. **Data Cleaning**: Before indexing, clean the CSV data to remove any unnecessary columns or rows that might be causing issues. Ensure that all columns have headers and that there are no empty rows. 4. **Review Chunking Settings**: Double-check the chunking settings in Dify to ensure they align with the structure of your CSV file. The default settings for the parent-child chunk strategy involve chunking by paragraphs and sentences, which might not be suitable for all CSV structures <sup>[[2]](https://github.com/langgenius/dify-docs/blob/main/en/guides/knowledge-base/create-knowledge-and-upload-documents/2.-choose-a-chunk-mode.md)</sup>. By addressing these areas, you may be able to resolve the issue with the CSV data turning into "Unnamed" and "nan" values during the indexing process. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/d7ba2cd0-91b6-4142-846a-7deabd627913?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/d7ba2cd0-91b6-4142-846a-7deabd627913?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/d7ba2cd0-91b6-4142-846a-7deabd627913?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/d7ba2cd0-91b6-4142-846a-7deabd627913?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/d7ba2cd0-91b6-4142-846a-7deabd627913?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/d7ba2cd0-91b6-4142-846a-7deabd627913?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/d7ba2cd0-91b6-4142-846a-7deabd627913?feedback_type=other)</sup>
yindo commented 2026-02-21 18:25:27 -05:00
Author
Owner
Copy Link
Copy Source

@crazywoola commented on GitHub (Feb 18, 2025):

I can not open this given csv file, seems to be a wrong format, please ensure the encoding is utf-8.

@crazywoola commented on GitHub (Feb 18, 2025): I can not open this given csv file, seems to be a wrong format, please ensure the encoding is `utf-8`.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#8391
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?