langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-22 04:25:29 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Embedding failed on v1.0.0 #8877

New Issue
Closed
opened 2026-02-21 18:28:18 -05:00 by yindo · 4 comments
yindo commented 2026-02-21 18:28:18 -05:00
Owner
Copy Link
Copy Source

Originally created by @zm1990s on GitHub (Mar 2, 2025).

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

v1.0.0

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

Document shows error state and retry will also fail. Unable to index.

✔️ Expected Behavior

No response

❌ Actual Behavior

Image

Backend API logs shows as follow:

2025-03-03 02:29:31.246 INFO [Dummy-6] [retry_document_indexing_task.py:57] - Start retry document: 7e2b6fe5-5267-42a0-92af-3d2b8479dac0
2025-03-03 02:29:31.317 ERROR [Dummy-6] [indexing_runner.py:96] - consume document failed
Traceback (most recent call last):
  File "/app/api/core/indexing_runner.py", line 73, in run
    documents = self._transform(
                ^^^^^^^^^^^^^^^^
  File "/app/api/core/indexing_runner.py", line 706, in _transform
    documents = index_processor.transform(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/rag/index_processor/processor/paragraph_index_processor.py", line 58, in transform
    document_nodes = splitter.split_documents([document])
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/rag/splitter/text_splitter.py", line 96, in split_documents
    return self.create_documents(texts, metadatas=metadatas)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/rag/splitter/text_splitter.py", line 81, in create_documents
    for chunk in self.split_text(text):
                 ^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/rag/splitter/fixed_text_splitter.py", line 68, in split_text
    chunks_lengths = self._length_function(chunks)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/rag/splitter/fixed_text_splitter.py", line 38, in _token_encoder
    return embedding_model_instance.get_text_embedding_num_tokens(texts=texts)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_manager.py", line 244, in get_text_embedding_num_tokens
    self._round_robin_invoke(
  File "/app/api/core/model_manager.py", line 370, in _round_robin_invoke
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 65, in get_num_tokens
    return plugin_model_manager.get_text_embedding_num_tokens(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/plugin/manager/model.py", line 313, in get_text_embedding_num_tokens
    for resp in response:
                ^^^^^^^^
  File "/app/api/core/plugin/manager/base.py", line 189, in _request_with_plugin_daemon_response_stream
    self._handle_plugin_daemon_error(error.error_type, error.message)
  File "/app/api/core/plugin/manager/base.py", line 221, in _handle_plugin_daemon_error
    raise PluginInvokeError(description=message)
core.plugin.manager.exc.PluginInvokeError: PluginInvokeError: {"error_type":"unmarshal_error","message":"unmarshal json failed: json: cannot unmarshal number into Go struct field GetTextEmbeddingNumTokensResponse.num_tokens of type []int"}
2025-03-03 02:29:31.319 INFO [Dummy-6] [retry_document_indexing_task.py:96] - Retry dataset: 9ff6bd6f-858d-4205-a581-c0c9fc7503b4 latency: 0.6457048403099179
Originally created by @zm1990s on GitHub (Mar 2, 2025). ### Self Checks - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:） - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version v1.0.0 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce Document shows error state and retry will also fail. Unable to index. ### ✔️ Expected Behavior _No response_ ### ❌ Actual Behavior <img width="1136" alt="Image" src="https://github.com/user-attachments/assets/da1b38b8-8e9f-428d-bb6f-92b31a653590" /> Backend API logs shows as follow: ``` 2025-03-03 02:29:31.246 INFO [Dummy-6] [retry_document_indexing_task.py:57] - Start retry document: 7e2b6fe5-5267-42a0-92af-3d2b8479dac0 2025-03-03 02:29:31.317 ERROR [Dummy-6] [indexing_runner.py:96] - consume document failed Traceback (most recent call last): File "/app/api/core/indexing_runner.py", line 73, in run documents = self._transform( ^^^^^^^^^^^^^^^^ File "/app/api/core/indexing_runner.py", line 706, in _transform documents = index_processor.transform( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/core/rag/index_processor/processor/paragraph_index_processor.py", line 58, in transform document_nodes = splitter.split_documents([document]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/core/rag/splitter/text_splitter.py", line 96, in split_documents return self.create_documents(texts, metadatas=metadatas) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/core/rag/splitter/text_splitter.py", line 81, in create_documents for chunk in self.split_text(text): ^^^^^^^^^^^^^^^^^^^^^ File "/app/api/core/rag/splitter/fixed_text_splitter.py", line 68, in split_text chunks_lengths = self._length_function(chunks) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/core/rag/splitter/fixed_text_splitter.py", line 38, in _token_encoder return embedding_model_instance.get_text_embedding_num_tokens(texts=texts) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/core/model_manager.py", line 244, in get_text_embedding_num_tokens self._round_robin_invoke( File "/app/api/core/model_manager.py", line 370, in _round_robin_invoke return function(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 65, in get_num_tokens return plugin_model_manager.get_text_embedding_num_tokens( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/core/plugin/manager/model.py", line 313, in get_text_embedding_num_tokens for resp in response: ^^^^^^^^ File "/app/api/core/plugin/manager/base.py", line 189, in _request_with_plugin_daemon_response_stream self._handle_plugin_daemon_error(error.error_type, error.message) File "/app/api/core/plugin/manager/base.py", line 221, in _handle_plugin_daemon_error raise PluginInvokeError(description=message) core.plugin.manager.exc.PluginInvokeError: PluginInvokeError: {"error_type":"unmarshal_error","message":"unmarshal json failed: json: cannot unmarshal number into Go struct field GetTextEmbeddingNumTokensResponse.num_tokens of type []int"} 2025-03-03 02:29:31.319 INFO [Dummy-6] [retry_document_indexing_task.py:96] - Retry dataset: 9ff6bd6f-858d-4205-a581-c0c9fc7503b4 latency: 0.6457048403099179 ```
yindo added the 🐞 bug👻 feat:rag labels 2026-02-21 18:28:18 -05:00
yindo closed this issue 2026-02-21 18:28:18 -05:00
yindo commented 2026-02-21 18:28:19 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Mar 2, 2025):

Hey there, @zm1990s! I'm here to help you with bug fixes, questions, and becoming a contributor. Let's squash those bugs together! 🐞

The embedding failure on Dify version v1.0.0 with a PluginInvokeError related to unmarshalling JSON is a known issue. This error occurs when using Tongyi Qianwen or Cohere embeddings due to a problem with the JSON structure, where a number is being incorrectly unmarshalled into a Go struct field. The issue has been fixed in the official Dify plugins. To resolve this problem, ensure you are using the latest version of Dify with the updated plugins that include the fix from PR #300 in the dify-official-plugins repository. Alternatively, you can switch your Dify installation to version v0.15.3 instead of using version 1.0.0 or the main branch code to avoid this error [1][2].

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other

@dosubot[bot] commented on GitHub (Mar 2, 2025): <!-- Greeting --> Hey there, @zm1990s! I'm here to help you with bug fixes, questions, and becoming a contributor. Let's squash those bugs together! 🐞 <!-- Answer --> The embedding failure on Dify version v1.0.0 with a `PluginInvokeError` related to unmarshalling JSON is a known issue. This error occurs when using Tongyi Qianwen or Cohere embeddings due to a problem with the JSON structure, where a number is being incorrectly unmarshalled into a Go struct field. The issue has been fixed in the official Dify plugins. To resolve this problem, ensure you are using the latest version of Dify with the updated plugins that include the fix from PR #300 in the dify-official-plugins repository. Alternatively, you can switch your Dify installation to version v0.15.3 instead of using version 1.0.0 or the main branch code to avoid this error <sup>[[1]](https://github.com/langgenius/dify/issues/14624)</sup><sup>[[2]](https://github.com/langgenius/dify/issues/14501)</sup>. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/7bc1fa3e-52de-4057-9db5-76d5f8f32d95?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/7bc1fa3e-52de-4057-9db5-76d5f8f32d95?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/7bc1fa3e-52de-4057-9db5-76d5f8f32d95?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/7bc1fa3e-52de-4057-9db5-76d5f8f32d95?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/7bc1fa3e-52de-4057-9db5-76d5f8f32d95?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/7bc1fa3e-52de-4057-9db5-76d5f8f32d95?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/7bc1fa3e-52de-4057-9db5-76d5f8f32d95?feedback_type=other)</sup>
yindo commented 2026-02-21 18:28:19 -05:00
Author
Owner
Copy Link
Copy Source

@zm1990s commented on GitHub (Mar 2, 2025):

Solved after updating Tongyi plugin to the latest version.

@zm1990s commented on GitHub (Mar 2, 2025): Solved after updating Tongyi plugin to the latest version.
yindo commented 2026-02-21 18:28:19 -05:00
Author
Owner
Copy Link
Copy Source

@xdmushui commented on GitHub (Mar 3, 2025):

Do you know the root cause?

@xdmushui commented on GitHub (Mar 3, 2025): Do you know the root cause?
yindo commented 2026-02-21 18:28:19 -05:00
Author
Owner
Copy Link
Copy Source

@meitianjinbu commented on GitHub (Mar 4, 2025):

Hey there, @zm1990s! I'm here to help you with bug fixes, questions, and becoming a contributor. Let's squash those bugs together! 🐞

The embedding failure on Dify version v1.0.0 with a PluginInvokeError related to unmarshalling JSON is a known issue. This error occurs when using Tongyi Qianwen or Cohere embeddings due to a problem with the JSON structure, where a number is being incorrectly unmarshalled into a Go struct field. The issue has been fixed in the official Dify plugins. To resolve this problem, ensure you are using the latest version of Dify with the updated plugins that include the fix from PR #300 in the dify-official-plugins repository. Alternatively, you can switch your Dify installation to version v0.15.3 instead of using version 1.0.0 or the main branch code to avoid this error [1][2].

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other

I meet another situation, embedding failed on v1.0.0 with the error info : can't start new thread. I have tried to low down the pool size and increase the nproc, Still not working！

Image

Image

@meitianjinbu commented on GitHub (Mar 4, 2025): > Hey there, [@zm1990s](https://github.com/zm1990s)! I'm here to help you with bug fixes, questions, and becoming a contributor. Let's squash those bugs together! 🐞 > > The embedding failure on Dify version v1.0.0 with a `PluginInvokeError` related to unmarshalling JSON is a known issue. This error occurs when using Tongyi Qianwen or Cohere embeddings due to a problem with the JSON structure, where a number is being incorrectly unmarshalled into a Go struct field. The issue has been fixed in the official Dify plugins. To resolve this problem, ensure you are using the latest version of Dify with the updated plugins that include the fix from PR [#300](https://github.com/langgenius/dify/pull/300) in the dify-official-plugins repository. Alternatively, you can switch your Dify installation to version v0.15.3 instead of using version 1.0.0 or the main branch code to avoid this error [[1]](https://github.com/langgenius/dify/issues/14624)[[2]](https://github.com/langgenius/dify/issues/14501). > > _To continue talking to [Dosu](https://dosu.dev), mention [@dosu](https://github.com/dosu)._ > > Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/7bc1fa3e-52de-4057-9db5-76d5f8f32d95?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/7bc1fa3e-52de-4057-9db5-76d5f8f32d95?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/7bc1fa3e-52de-4057-9db5-76d5f8f32d95?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/7bc1fa3e-52de-4057-9db5-76d5f8f32d95?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/7bc1fa3e-52de-4057-9db5-76d5f8f32d95?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/7bc1fa3e-52de-4057-9db5-76d5f8f32d95?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/7bc1fa3e-52de-4057-9db5-76d5f8f32d95?feedback_type=other) I meet another situation, embedding failed on v1.0.0 with the error info : can't start new thread. I have tried to low down the pool size and increase the nproc, Still not working！ ![Image](https://github.com/user-attachments/assets/1cf8265d-2c52-4316-8531-ce94d469d09d) ![Image](https://github.com/user-attachments/assets/c7a3cdc7-7815-44ac-a61d-bda74c4892ad)
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug 👻 feat:rag
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#8877
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?