langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-23 04:55:28 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Text cannot be indexed using MyScale #5030

New Issue
Closed
opened 2026-02-21 18:09:05 -05:00 by yindo · 1 comment
yindo commented 2026-02-21 18:09:05 -05:00
Owner
Copy Link
Copy Source

Originally created by @MochiXu on GitHub (Aug 14, 2024).

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

master

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

✔️ Expected Behavior

No response

❌ Actual Behavior

No response

Originally created by @MochiXu on GitHub (Aug 14, 2024). ### Self Checks - [X] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [X] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [X] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [X] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:） - [X] Please do not modify this template :) and fill in all the required fields. ### Dify version master ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce ![](https://cdn.discordapp.com/attachments/1082908561597796352/1272494937417453578/Screenshot_20.png?ex=66bd2902&is=66bbd782&hm=d135fcb4e02ecfeac628470eddf10eac098c69ecf30496b871aae620d527984c&) ### ✔️ Expected Behavior _No response_ ### ❌ Actual Behavior _No response_
yindo added the 🐞 bug label 2026-02-21 18:09:05 -05:00
yindo closed this issue 2026-02-21 18:09:05 -05:00
yindo commented 2026-02-21 18:09:05 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Aug 14, 2024):

Hey @MochiXu! 👋 I'm here to help you out with bugs, questions, and becoming a contributor. Let's squash those bugs together!

To resolve the issue where text cannot be indexed using MyScale in the Dify application, resulting in an embedding completion error, follow these steps:

  1. Choose the Appropriate Indexing Method:

    • High-Quality Mode: Uses OpenAI's embedding interface for higher accuracy.
    • Economy Mode: Uses keyword indexing, which is less accurate but does not incur token costs.
    • Q&A Mode: Matches questions directly to questions for precise information retrieval.

  2. Configure Retrieval Settings:

    • Vector Search: Generates query embeddings to find the most similar text chunk.
    • Full-Text Search: Indexes all terms in the document for term-based retrieval.
    • Hybrid Search: Combines full-text and vector searches, re-ranking results for the best match.

  3. Ensure Proper Segmentation and Cleaning:

    • Segment the text appropriately to match the context window of the language model.
    • Clean the text to remove unwanted characters or blank lines that might affect the quality of the response. For example, you can enable rules like removing extra spaces, URLs, emails, and stopwords.

  4. Check ETL Configuration:

    • Ensure that the ETL (extract, transform, load) process is correctly set up to preprocess and clean the data.

  5. Verify Upload Limitations:

    • Ensure the document size does not exceed 15MB and the batch upload does not exceed 20 files.

  6. Check Rate Limits:

    • Ensure that the rate limit has not been reached for the Embedding model API key used. If the rate limit is reached, you may need to wait or upgrade your plan.

  7. Monitor Indexing Status:

    • Use the Dify application's indexing status monitoring features to track the progress of the embedding process. Ensure that the status is not stuck in 'indexing', 'splitting', 'parsing', or 'cleaning' stages. If errors occur, check the detailed error messages for specific issues.

  8. Pause and Resume Indexing:

    • If the indexing process is stuck, you can use the Dify application's feature to pause and resume the indexing process. This can be done using the pauseDocIndexing and resumeDocIndexing functions provided by the Dify service [1][2][3][4].

By following these steps, you can address the embedding completion error and successfully index the text using MyScale in the Dify application.

To continue talking to Dosu, mention @dosu.

@dosubot[bot] commented on GitHub (Aug 14, 2024): <!-- Greeting --> Hey @MochiXu! 👋 I'm here to help you out with bugs, questions, and becoming a contributor. Let's squash those bugs together! <!-- Answer --> To resolve the issue where text cannot be indexed using MyScale in the Dify application, resulting in an embedding completion error, follow these steps: 1. **Choose the Appropriate Indexing Method**: - **High-Quality Mode**: Uses OpenAI's embedding interface for higher accuracy. - **Economy Mode**: Uses keyword indexing, which is less accurate but does not incur token costs. - **Q&A Mode**: Matches questions directly to questions for precise information retrieval. 2. **Configure Retrieval Settings**: - **Vector Search**: Generates query embeddings to find the most similar text chunk. - **Full-Text Search**: Indexes all terms in the document for term-based retrieval. - **Hybrid Search**: Combines full-text and vector searches, re-ranking results for the best match. 3. **Ensure Proper Segmentation and Cleaning**: - Segment the text appropriately to match the context window of the language model. - Clean the text to remove unwanted characters or blank lines that might affect the quality of the response. For example, you can enable rules like removing extra spaces, URLs, emails, and stopwords. 4. **Check ETL Configuration**: - Ensure that the ETL (extract, transform, load) process is correctly set up to preprocess and clean the data. 5. **Verify Upload Limitations**: - Ensure the document size does not exceed 15MB and the batch upload does not exceed 20 files. 6. **Check Rate Limits**: - Ensure that the rate limit has not been reached for the Embedding model API key used. If the rate limit is reached, you may need to wait or upgrade your plan. 7. **Monitor Indexing Status**: - Use the Dify application's indexing status monitoring features to track the progress of the embedding process. Ensure that the status is not stuck in 'indexing', 'splitting', 'parsing', or 'cleaning' stages. If errors occur, check the detailed error messages for specific issues. 8. **Pause and Resume Indexing**: - If the indexing process is stuck, you can use the Dify application's feature to pause and resume the indexing process. This can be done using the `pauseDocIndexing` and `resumeDocIndexing` functions provided by the Dify service <sup>[[1]](https://github.com/langgenius/dify-docs/blob/main/en/guides/knowledge-base/create-knowledge-and-upload-documents.md)</sup><sup>[[2]](https://github.com/langgenius/dify-docs/blob/main/en/guides/application-orchestrate/llms-use-faq.md)</sup><sup>[[3]](https://github.com/langgenius/dify/blob/main/web/app/components/datasets/create/embedding-process/index.tsx)</sup><sup>[[4]](https://github.com/langgenius/dify/blob/main/web/app/components/datasets/documents/detail/embedding/index.tsx)</sup>. By following these steps, you can address the embedding completion error and successfully index the text using MyScale in the Dify application. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#5030
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?