langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 12:05:30 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

When upload many texts( >30,000 ) for knowledge base using api, it cost too much times to be ready. #7947

New Issue
Closed
opened 2026-02-21 18:23:10 -05:00 by yindo · 9 comments
yindo commented 2026-02-21 18:23:10 -05:00
Owner
Copy Link
Copy Source

Originally created by @shawn-maxiao on GitHub (Feb 1, 2025).

Originally assigned to: @JohnJyong on GitHub.

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

:0.15.0

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

I have red the source to find why. And I found dify used much times to extract the keywords.
I found the extract-keywords function uses Jieba to extract keywords and save the keyword tables. The save function very slow( about 2~5 seconds to save dataset keyword table for one document chunk).
self._save_dataset_keyword_table(keyword_table)

And another big problem is there is one redis lock, which makes the other extract-keywords threads wait for the extract-keywords thread which is executing, this causes the process looks like one-by-one instead of parallel.
lock_name = "keyword_indexing_lock_{}".format(self.dataset.id) with redis_client.lock(lock_name, timeout=600): ......

How to solve it?

✔️ Expected Behavior

No response

❌ Actual Behavior

No response

Originally created by @shawn-maxiao on GitHub (Feb 1, 2025). Originally assigned to: @JohnJyong on GitHub. ### Self Checks - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:） - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version :0.15.0 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce I have red the source to find why. And I found dify used much times to extract the keywords. I found the extract-keywords function uses Jieba to extract keywords and save the keyword tables. The save function very slow( about 2~5 seconds to save dataset keyword table for one document chunk). ` self._save_dataset_keyword_table(keyword_table) ` And another big problem is there is one redis lock, which makes the other extract-keywords threads wait for the extract-keywords thread which is executing, this causes the process looks like one-by-one instead of parallel. `lock_name = "keyword_indexing_lock_{}".format(self.dataset.id) with redis_client.lock(lock_name, timeout=600): ...... ` How to solve it? ### ✔️ Expected Behavior _No response_ ### ❌ Actual Behavior _No response_
yindo added the 🐞 bugperformance labels 2026-02-21 18:23:10 -05:00
yindo closed this issue 2026-02-21 18:23:10 -05:00
yindo commented 2026-02-21 18:23:11 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Feb 1, 2025):

@shawn-maxiao Hello there! I'm here to assist you with any bugs, questions, or contributions. Let's tackle this issue together while we wait for a human maintainer.

To address the performance issues you're experiencing with the keyword extraction function in Dify, especially when dealing with a large number of texts, you can consider the following optimizations:

  1. Remove Redis Lock for Parallel Processing: The Redis lock is causing sequential processing, which can be a bottleneck. By removing the lock, you can enable parallel processing of texts. This can be achieved by processing each text in a separate thread or process, which will significantly improve performance.

  2. Batch Processing: Instead of processing each text individually, you can batch multiple texts together and process them in a single operation. This reduces the overhead of acquiring and releasing locks and can improve the efficiency of keyword extraction.

  3. Optimize Jieba Usage: Consider optimizing the usage of Jieba for keyword extraction. This might involve tuning parameters or using a more efficient algorithm or library for keyword extraction.

  4. Asynchronous Processing: Implement asynchronous processing to handle keyword extraction, which can help utilize the CPU more efficiently and lead to better performance.

Here's a conceptual example of how you might implement parallel processing by removing the Redis lock:

from concurrent.futures import ThreadPoolExecutor

def process_text(text, keyword_table_handler, config, dataset_id):
    keywords = keyword_table_handler.extract_keywords(text.page_content, config.max_keywords_per_chunk)
    if text.metadata is not None:
        _update_segment_keywords(dataset_id, text.metadata["doc_id"], list(keywords))
        return text.metadata["doc_id"], list(keywords)
    return None, None

def add_texts_parallel(self, texts: list[Document], **kwargs):
    keyword_table_handler = JiebaKeywordTableHandler()
    keyword_table = self._get_dataset_keyword_table()
    keywords_list = kwargs.get("keywords_list")

    with ThreadPoolExecutor() as executor:
        futures = []
        for i in range(len(texts)):
            text = texts[i]
            if keywords_list:
                keywords = keywords_list[i]
                if not keywords:
                    keywords = keyword_table_handler.extract_keywords(
                        text.page_content, self._config.max_keywords_per_chunk
                    )
            else:
                keywords = keyword_table_handler.extract_keywords(
                    text.page_content, self._config.max_keywords_per_chunk
                )
            futures.append(executor.submit(process_text, text, keyword_table_handler, self._config, self.dataset.id))

        for future in futures:
            doc_id, keywords = future.result()
            if doc_id and keywords:
                keyword_table = self._add_text_to_keyword_table(
                    keyword_table or {}, doc_id, keywords
                )

    self._save_dataset_keyword_table(keyword_table)

This example uses a ThreadPoolExecutor to process texts in parallel, which can improve performance by utilizing multiple CPU cores. Note that this is a conceptual example and may need adjustments based on your specific environment and requirements [1].

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other

@dosubot[bot] commented on GitHub (Feb 1, 2025): <!-- Greeting --> @shawn-maxiao Hello there! I'm here to assist you with any bugs, questions, or contributions. Let's tackle this issue together while we wait for a human maintainer. <!-- Answer --> To address the performance issues you're experiencing with the keyword extraction function in Dify, especially when dealing with a large number of texts, you can consider the following optimizations: 1. **Remove Redis Lock for Parallel Processing**: The Redis lock is causing sequential processing, which can be a bottleneck. By removing the lock, you can enable parallel processing of texts. This can be achieved by processing each text in a separate thread or process, which will significantly improve performance. 2. **Batch Processing**: Instead of processing each text individually, you can batch multiple texts together and process them in a single operation. This reduces the overhead of acquiring and releasing locks and can improve the efficiency of keyword extraction. 3. **Optimize Jieba Usage**: Consider optimizing the usage of Jieba for keyword extraction. This might involve tuning parameters or using a more efficient algorithm or library for keyword extraction. 4. **Asynchronous Processing**: Implement asynchronous processing to handle keyword extraction, which can help utilize the CPU more efficiently and lead to better performance. Here's a conceptual example of how you might implement parallel processing by removing the Redis lock: ```python from concurrent.futures import ThreadPoolExecutor def process_text(text, keyword_table_handler, config, dataset_id): keywords = keyword_table_handler.extract_keywords(text.page_content, config.max_keywords_per_chunk) if text.metadata is not None: _update_segment_keywords(dataset_id, text.metadata["doc_id"], list(keywords)) return text.metadata["doc_id"], list(keywords) return None, None def add_texts_parallel(self, texts: list[Document], **kwargs): keyword_table_handler = JiebaKeywordTableHandler() keyword_table = self._get_dataset_keyword_table() keywords_list = kwargs.get("keywords_list") with ThreadPoolExecutor() as executor: futures = [] for i in range(len(texts)): text = texts[i] if keywords_list: keywords = keywords_list[i] if not keywords: keywords = keyword_table_handler.extract_keywords( text.page_content, self._config.max_keywords_per_chunk ) else: keywords = keyword_table_handler.extract_keywords( text.page_content, self._config.max_keywords_per_chunk ) futures.append(executor.submit(process_text, text, keyword_table_handler, self._config, self.dataset.id)) for future in futures: doc_id, keywords = future.result() if doc_id and keywords: keyword_table = self._add_text_to_keyword_table( keyword_table or {}, doc_id, keywords ) self._save_dataset_keyword_table(keyword_table) ``` This example uses a `ThreadPoolExecutor` to process texts in parallel, which can improve performance by utilizing multiple CPU cores. Note that this is a conceptual example and may need adjustments based on your specific environment and requirements <sup>[[1]](https://github.com/langgenius/dify/blob/main/api/core/rag/datasource/keyword/jieba/jieba.py)</sup>. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/acea8045-69b9-439e-96ce-dd3c88bd6898?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/acea8045-69b9-439e-96ce-dd3c88bd6898?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/acea8045-69b9-439e-96ce-dd3c88bd6898?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/acea8045-69b9-439e-96ce-dd3c88bd6898?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/acea8045-69b9-439e-96ce-dd3c88bd6898?feedback_type=hallucination) | [Other](https://app.dosu.dev/response-feedback/acea8045-69b9-439e-96ce-dd3c88bd6898?feedback_type=other)</sup>
yindo commented 2026-02-21 18:23:11 -05:00
Author
Owner
Copy Link
Copy Source

@shawn-maxiao commented on GitHub (Feb 8, 2025):

Could I disable the keywords extract function to avoid the problem in dify configuration?

@shawn-maxiao commented on GitHub (Feb 8, 2025): Could I disable the keywords extract function to avoid the problem in dify configuration?
yindo commented 2026-02-21 18:23:11 -05:00
Author
Owner
Copy Link
Copy Source

@j0rGeT commented on GitHub (Feb 26, 2025):

same problem, how to resolve?

@j0rGeT commented on GitHub (Feb 26, 2025): same problem, how to resolve?
yindo commented 2026-02-21 18:23:11 -05:00
Author
Owner
Copy Link
Copy Source

@shawn-maxiao commented on GitHub (Feb 27, 2025):

same problem, how to resolve?
Just change the llmops platform from dify to ragflow/...

@shawn-maxiao commented on GitHub (Feb 27, 2025): > same problem, how to resolve? Just change the llmops platform from dify to ragflow/...
yindo commented 2026-02-21 18:23:11 -05:00
Author
Owner
Copy Link
Copy Source

@imaben commented on GitHub (Mar 11, 2025):

same problem!!

@imaben commented on GitHub (Mar 11, 2025): same problem!!
yindo commented 2026-02-21 18:23:11 -05:00
Author
Owner
Copy Link
Copy Source

@imaben commented on GitHub (Mar 11, 2025):

@JohnJyong
This issue has persisted for a very long time and has a significant impact on the indexing efficiency of knowledge bases with a large number of documents. Please pay close attention to it. Thank you very much.

@imaben commented on GitHub (Mar 11, 2025): @JohnJyong This issue has persisted for a very long time and has a significant impact on the indexing efficiency of knowledge bases with a large number of documents. Please pay close attention to it. Thank you very much.
yindo commented 2026-02-21 18:23:11 -05:00
Author
Owner
Copy Link
Copy Source

@imaben commented on GitHub (Mar 13, 2025):

@JohnJyong
Urgently seeking a solution, or to put it another way, is there a temporary method to block it? Orz！

@imaben commented on GitHub (Mar 13, 2025): @JohnJyong Urgently seeking a solution, or to put it another way, is there a temporary method to block it? Orz！
yindo commented 2026-02-21 18:23:11 -05:00
Author
Owner
Copy Link
Copy Source

@dongjinhai commented on GitHub (Mar 25, 2025):

same problem

@dongjinhai commented on GitHub (Mar 25, 2025): same problem
yindo commented 2026-02-21 18:23:11 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Apr 26, 2025):

Hi, @shawn-maxiao. I'm Dosu, and I'm helping the Dify team manage their backlog. I'm marking this issue as stale.

Issue Summary:

  • You reported slow performance when uploading over 30,000 texts due to sequential keyword extraction using Jieba and a Redis lock.
  • I suggested optimizations like removing the Redis lock for parallel processing and batch processing.
  • Other users, including j0rGeT, imaben, and dongjinhai, have experienced similar issues.
  • Imaben urged maintainer JohnJyong for a solution or workaround.
  • You suggested switching from the Dify platform to ragflow as a potential resolution.

Next Steps:

  • Please let us know if this issue is still relevant to the latest version of the Dify repository by commenting here.
  • If no updates are provided, this issue will be automatically closed in 15 days.

Thank you for your understanding and contribution!

@dosubot[bot] commented on GitHub (Apr 26, 2025): Hi, @shawn-maxiao. I'm [Dosu](https://dosu.dev), and I'm helping the Dify team manage their backlog. I'm marking this issue as stale. **Issue Summary:** - You reported slow performance when uploading over 30,000 texts due to sequential keyword extraction using Jieba and a Redis lock. - I suggested optimizations like removing the Redis lock for parallel processing and batch processing. - Other users, including j0rGeT, imaben, and dongjinhai, have experienced similar issues. - Imaben urged maintainer JohnJyong for a solution or workaround. - You suggested switching from the Dify platform to ragflow as a potential resolution. **Next Steps:** - Please let us know if this issue is still relevant to the latest version of the Dify repository by commenting here. - If no updates are provided, this issue will be automatically closed in 15 days. Thank you for your understanding and contribution!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label performance 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#7947
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?