langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 12:05:30 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

new document embedding error, PluginlnvokeError: {"args":0,"error_type":"HTTPStatusError","message":"Client error '413Payload Too Large' for url'http://xx.xxx.xx.xx:xxxx/tokenize\nFor more information #9393

New Issue
Closed
opened 2026-02-21 18:34:11 -05:00 by yindo · 0 comments
yindo commented 2026-02-21 18:34:11 -05:00
Owner
Copy Link
Copy Source

Originally created by @Thunderinmoonlight on GitHub (Mar 5, 2025).

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

1.0.0

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

I use TEI for embeding inference，An error is reported in the new document embedding,Documents on completed embedding can be recalled

worker-1         | 2025-03-05 16:46:30.130 ERROR [Dummy-12] [indexing_runner.py:96] - consume document failed
worker-1         | Traceback (most recent call last):
worker-1         |   File "/app/api/core/indexing_runner.py", line 73, in run
worker-1         |     documents = self._transform(
worker-1         |                 ^^^^^^^^^^^^^^^^
worker-1         |   File "/app/api/core/indexing_runner.py", line 706, in _transform
worker-1         |     documents = index_processor.transform(
worker-1         |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
worker-1         |   File "/app/api/core/rag/index_processor/processor/paragraph_index_processor.py", line 58, in transform
worker-1         |     document_nodes = splitter.split_documents([document])
worker-1         |                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker-1         |   File "/app/api/core/rag/splitter/text_splitter.py", line 96, in split_documents
worker-1         |     return self.create_documents(texts, metadatas=metadatas)
worker-1         |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker-1         |   File "/app/api/core/rag/splitter/text_splitter.py", line 81, in create_documents
worker-1         |     for chunk in self.split_text(text):
worker-1         |                  ^^^^^^^^^^^^^^^^^^^^^
worker-1         |   File "/app/api/core/rag/splitter/fixed_text_splitter.py", line 71, in split_text
worker-1         |     final_chunks.extend(self.recursive_split_text(chunk))
worker-1         |                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker-1         |   File "/app/api/core/rag/splitter/fixed_text_splitter.py", line 97, in recursive_split_text
worker-1         |     s_lens = self._length_function(splits)
worker-1         |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker-1         |   File "/app/api/core/rag/splitter/fixed_text_splitter.py", line 38, in _token_encoder
worker-1         |     return embedding_model_instance.get_text_embedding_num_tokens(texts=texts)
worker-1         |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker-1         |   File "/app/api/core/model_manager.py", line 244, in get_text_embedding_num_tokens
worker-1         |     self._round_robin_invoke(
worker-1         |   File "/app/api/core/model_manager.py", line 370, in _round_robin_invoke
worker-1         |     return function(*args, **kwargs)
worker-1         |            ^^^^^^^^^^^^^^^^^^^^^^^^^
worker-1         |   File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 65, in get_num_tokens
worker-1         |     return plugin_model_manager.get_text_embedding_num_tokens(
worker-1         |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker-1         |   File "/app/api/core/plugin/manager/model.py", line 313, in get_text_embedding_num_tokens
worker-1         |     for resp in response:
worker-1         |                 ^^^^^^^^
worker-1         |   File "/app/api/core/plugin/manager/base.py", line 189, in _request_with_plugin_daemon_response_stream
worker-1         |     self._handle_plugin_daemon_error(error.error_type, error.message)
worker-1         |   File "/app/api/core/plugin/manager/base.py", line 221, in _handle_plugin_daemon_error
worker-1         |     raise PluginInvokeError(description=message)
worker-1         | core.plugin.manager.exc.PluginInvokeError: PluginInvokeError: {"args":{},"error_type":"HTTPStatusError","message":"Client error '413 Payload Too Large' for url 'http://xxxxx:xxx/tokenize'\nFor more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/413"}

Image

✔️ Expected Behavior

No response

❌ Actual Behavior

No response

Originally created by @Thunderinmoonlight on GitHub (Mar 5, 2025). ### Self Checks - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:） - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version 1.0.0 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce I use TEI for embeding inference，An error is reported in the new document embedding,Documents on completed embedding can be recalled ``` worker-1 | 2025-03-05 16:46:30.130 ERROR [Dummy-12] [indexing_runner.py:96] - consume document failed worker-1 | Traceback (most recent call last): worker-1 | File "/app/api/core/indexing_runner.py", line 73, in run worker-1 | documents = self._transform( worker-1 | ^^^^^^^^^^^^^^^^ worker-1 | File "/app/api/core/indexing_runner.py", line 706, in _transform worker-1 | documents = index_processor.transform( worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^ worker-1 | File "/app/api/core/rag/index_processor/processor/paragraph_index_processor.py", line 58, in transform worker-1 | document_nodes = splitter.split_documents([document]) worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker-1 | File "/app/api/core/rag/splitter/text_splitter.py", line 96, in split_documents worker-1 | return self.create_documents(texts, metadatas=metadatas) worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker-1 | File "/app/api/core/rag/splitter/text_splitter.py", line 81, in create_documents worker-1 | for chunk in self.split_text(text): worker-1 | ^^^^^^^^^^^^^^^^^^^^^ worker-1 | File "/app/api/core/rag/splitter/fixed_text_splitter.py", line 71, in split_text worker-1 | final_chunks.extend(self.recursive_split_text(chunk)) worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker-1 | File "/app/api/core/rag/splitter/fixed_text_splitter.py", line 97, in recursive_split_text worker-1 | s_lens = self._length_function(splits) worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker-1 | File "/app/api/core/rag/splitter/fixed_text_splitter.py", line 38, in _token_encoder worker-1 | return embedding_model_instance.get_text_embedding_num_tokens(texts=texts) worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker-1 | File "/app/api/core/model_manager.py", line 244, in get_text_embedding_num_tokens worker-1 | self._round_robin_invoke( worker-1 | File "/app/api/core/model_manager.py", line 370, in _round_robin_invoke worker-1 | return function(*args, **kwargs) worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^ worker-1 | File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 65, in get_num_tokens worker-1 | return plugin_model_manager.get_text_embedding_num_tokens( worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker-1 | File "/app/api/core/plugin/manager/model.py", line 313, in get_text_embedding_num_tokens worker-1 | for resp in response: worker-1 | ^^^^^^^^ worker-1 | File "/app/api/core/plugin/manager/base.py", line 189, in _request_with_plugin_daemon_response_stream worker-1 | self._handle_plugin_daemon_error(error.error_type, error.message) worker-1 | File "/app/api/core/plugin/manager/base.py", line 221, in _handle_plugin_daemon_error worker-1 | raise PluginInvokeError(description=message) worker-1 | core.plugin.manager.exc.PluginInvokeError: PluginInvokeError: {"args":{},"error_type":"HTTPStatusError","message":"Client error '413 Payload Too Large' for url 'http://xxxxx:xxx/tokenize'\nFor more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/413"} ``` ![Image](https://github.com/user-attachments/assets/8d9e1097-33dd-46ef-887d-248135a2bdcb) ### ✔️ Expected Behavior _No response_ ### ❌ Actual Behavior _No response_
yindo added the 🐞 bug👻 feat:rag labels 2026-02-21 18:34:11 -05:00
yindo closed this issue 2026-02-21 18:34:11 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug 👻 feat:rag
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#9393
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?