langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-20 19:50:06 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Dify Sending Empty Embedding Requests #4831

New Issue
Closed
opened 2026-02-21 18:08:12 -05:00 by yindo · 5 comments
yindo commented 2026-02-21 18:08:12 -05:00
Owner
Copy Link
Copy Source

Originally created by @mbbyn on GitHub (Jul 30, 2024).

Originally assigned to: @laipz8200, @JohnJyong, @crazywoola on GitHub.

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

0.6.15

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

I don't know what the exact parameters are to reproduce the problem, but I can reproduce 100% of the time using this:

  1. Setup an OpenAI API compatible server, such as HuggingFace TEI
  2. Upload DOCX file and set chunk size to 2000

It also happens to other formats, it seems to be more likely to happen with larger chunk sizes.

✔️ Expected Behavior

It should properly chunk and create embeddings

❌ Actual Behavior

It fails due to sending an empty embeddings request to the embeddings server, causing the whole process to fail.

References

NOTE: The time difference between the logs is because of time drift on each server, they are not on the same server.

HF TEI Logs

{
    "timestamp": "2024-07-30T12:32:16.754942Z",
    "level": "INFO",
    "message": "Success",
    "target": "text_embeddings_router::http::server",
    "filename": "router/src/http/server.rs",
    "line_number": 1246,
    "span": {
        "inference_time": "13.135229ms",
        "queue_time": "703.432µs",
        "tokenization_time": "627.369µs",
        "total_time": "14.607593ms",
        "name": "openai_embed"
    },
    "spans": [
        {
            "inference_time": "13.135229ms",
            "queue_time": "703.432µs",
            "tokenization_time": "627.369µs",
            "total_time": "14.607593ms",
            "name": "openai_embed"
        }
    ]
}
 {
    "timestamp": "2024-07-30T12:32:16.763780Z",
    "level": "ERROR",
    "message": "Input validation error: `inputs` cannot be empty",
    "target": "text_embeddings_core::infer",
    "filename": "core/src/infer.rs",
    "line_number": 332,
    "span": {
        "normalize": true,
        "prompt_name": "None",
        "truncate": false,
        "truncation_direction": "Right",
        "name": "embed_pooled"
    },
    "spans": [
        {
            "name": "openai_embed"
        },
        {
            "normalize": true,
            "prompt_name": "None",
            "truncate": false,
            "truncation_direction": "Right",
            "name": "embed_pooled"
        }
    ]
}
 {
    "timestamp": "2024-07-30T12:33:33.908732Z",
    "level": "ERROR",
    "message": "Input validation error: `inputs` cannot be empty",
    "target": "text_embeddings_core::infer",
    "filename": "core/src/infer.rs",
    "line_number": 332,
    "span": {
        "normalize": true,
        "prompt_name": "None",
        "truncate": false,
        "truncation_direction": "Right",
        "name": "embed_pooled"
    },
    "spans": [
        {
            "name": "openai_embed"
        },
        {
            "normalize": true,
            "prompt_name": "None",
            "truncate": false,
            "truncation_direction": "Right",
            "name": "embed_pooled"
        }
    ]
}

Reproduced from the REPL

>>> client.embeddings.create(input="blahblabh"*100000, model="sentence-transformers/distiluse-base-multilingual-cased-v1")
Traceback (most recent call last):
  File "<input>", line 1, in <module>
    client.embeddings.create(input="blahblabh"*100000, model="sentence-transformers/distiluse-base-multilingual-cased-v1")
  File "~/.cache/pypoetry/virtualenvs/poc-slot-machine-DH_FKYDt-py3.11/lib/python3.11/site-packages/openai/resources/embeddings.py", line 114, in create
    return self._post(
           ^^^^^^^^^^^
  File "~/.cache/pypoetry/virtualenvs/poc-slot-machine-DH_FKYDt-py3.11/lib/python3.11/site-packages/openai/_base_client.py", line 1266, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.cache/pypoetry/virtualenvs/poc-slot-machine-DH_FKYDt-py3.11/lib/python3.11/site-packages/openai/_base_client.py", line 942, in request
    return self._request(
           ^^^^^^^^^^^^^^
  File "~/.cache/pypoetry/virtualenvs/poc-slot-machine-DH_FKYDt-py3.11/lib/python3.11/site-packages/openai/_base_client.py", line 1046, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.APIStatusError: Error code: 413 - {'message': 'Input validation error: `inputs` must have less than 128000 characters. Given: 900000', 'code': 413, 'type': 'Validation'}
>>> client.embeddings.create(input="", model="sentence-transformers/distiluse-base-multilingual-cased-v1")
Traceback (most recent call last):
  File "<input>", line 1, in <module>
    client.embeddings.create(input="", model="sentence-transformers/distiluse-base-multilingual-cased-v1")
  File "~/.cache/pypoetry/virtualenvs/poc-slot-machine-DH_FKYDt-py3.11/lib/python3.11/site-packages/openai/resources/embeddings.py", line 114, in create
    return self._post(
           ^^^^^^^^^^^
  File "~/.cache/pypoetry/virtualenvs/poc-slot-machine-DH_FKYDt-py3.11/lib/python3.11/site-packages/openai/_base_client.py", line 1266, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.cache/pypoetry/virtualenvs/poc-slot-machine-DH_FKYDt-py3.11/lib/python3.11/site-packages/openai/_base_client.py", line 942, in request
    return self._request(
           ^^^^^^^^^^^^^^
  File "~/.cache/pypoetry/virtualenvs/poc-slot-machine-DH_FKYDt-py3.11/lib/python3.11/site-packages/openai/_base_client.py", line 1046, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.APIStatusError: Error code: 413 - {'message': 'Input validation error: `inputs` cannot be empty', 'code': 413, 'type': 'Validation'}

Dify Worker Logs

⚠️ Please don't pay attention to "413 Payload too large", this is not the real error. As you can see in the TEI server, the error code is indeed 413, but the actual error is that Dify is sending empty embeddings request.

 [2024-07-30 12:32:31,515: INFO/MainProcess] Task tasks.clean_dataset_task.clean_dataset_task[a766319d-055b-4ed1-8661-a5d31cebfb1e] received
 [2024-07-30 12:32:31,516: INFO/MainProcess] Start clean dataset when dataset deleted: 6c33868a-5b2d-4e4f-9b5e-1deebb7ec3d8
 [2024-07-30 12:32:31,520: INFO/MainProcess] Cleaning documents for dataset: 6c33868a-5b2d-4e4f-9b5e-1deebb7ec3d8
 [2024-07-30 12:32:31,585: INFO/MainProcess] Cleaned dataset when dataset deleted: 6c33868a-5b2d-4e4f-9b5e-1deebb7ec3d8 latency: 0.0696456499863416
 [2024-07-30 12:32:31,612: INFO/MainProcess] Task tasks.clean_dataset_task.clean_dataset_task[a766319d-055b-4ed1-8661-a5d31cebfb1e] succeeded in 0.09655854594893754s: None
 [2024-07-30 12:33:22,548: INFO/MainProcess] Task tasks.document_indexing_task.document_indexing_task[1aebb6c4-b365-47df-9599-0b3b294742e5] received
 [2024-07-30 12:33:22,552: INFO/MainProcess] Start process document: 25a7ffea-c071-47a2-b394-982ba6eb56a0
 [2024-07-30 12:33:22,692: WARNING/MainProcess] --- Logging error ---
 [2024-07-30 12:33:22,692: WARNING/MainProcess] Traceback (most recent call last):
 [2024-07-30 12:33:22,692: WARNING/MainProcess]   File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 36, in invoke
     return self._invoke(model, credentials, texts, user)
 [2024-07-30 12:33:22,692: WARNING/MainProcess]   File "/app/api/core/model_runtime/model_providers/openai_api_compatible/text_embedding/text_embedding.py", line 105, in _invoke
     response.raise_for_status()  # Raise an exception for HTTP errors
 [2024-07-30 12:33:22,692: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
     raise HTTPError(http_error_msg, response=self)
 [2024-07-30 12:33:22,693: WARNING/MainProcess] requests.exceptions.HTTPError: 413 Client Error: Payload Too Large for url: http://[MASKED]/v1/embeddings
 [2024-07-30 12:33:22,693: WARNING/MainProcess]
 During handling of the above exception, another exception occurred:
 [2024-07-30 12:33:22,693: WARNING/MainProcess] Traceback (most recent call last):
 [2024-07-30 12:33:22,693: WARNING/MainProcess]   File "/app/api/core/embedding/cached_embedding.py", line 51, in embed_documents
     embedding_result = self._model_instance.invoke_text_embedding(
 [2024-07-30 12:33:22,693: WARNING/MainProcess]   File "/app/api/core/model_manager.py", line 170, in invoke_text_embedding
     return self._round_robin_invoke(
 [2024-07-30 12:33:22,693: WARNING/MainProcess]   File "/app/api/core/model_manager.py", line 302, in _round_robin_invoke
     return function(*args, **kwargs)
 [2024-07-30 12:33:22,693: WARNING/MainProcess]   File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 38, in invoke
     raise self._transform_invoke_error(e)
 [2024-07-30 12:33:22,693: WARNING/MainProcess] core.model_runtime.errors.invoke.InvokeBadRequestError: [openai_api_compatible] Bad Request Error, 413 Client Error: Payload Too Large for url: http://[MASKED]/v1/embeddings
 [2024-07-30 12:33:22,693: WARNING/MainProcess]
 During handling of the above exception, another exception occurred:
 [2024-07-30 12:33:22,693: WARNING/MainProcess] Traceback (most recent call last):
 [2024-07-30 12:33:22,693: WARNING/MainProcess]   File "/usr/local/lib/python3.10/logging/__init__.py", line 1100, in emit
     msg = self.format(record)
 [2024-07-30 12:33:22,693: WARNING/MainProcess]   File "/usr/local/lib/python3.10/logging/__init__.py", line 943, in format
     return fmt.format(record)
 [2024-07-30 12:33:22,693: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/celery/utils/log.py", line 146, in format
     msg = super().format(record)
 [2024-07-30 12:33:22,693: WARNING/MainProcess]   File "/usr/local/lib/python3.10/logging/__init__.py", line 678, in format
     record.message = record.getMessage()
 [2024-07-30 12:33:22,694: WARNING/MainProcess]   File "/usr/local/lib/python3.10/logging/__init__.py", line 368, in getMessage
     msg = msg % self.args
 [2024-07-30 12:33:22,694: WARNING/MainProcess] TypeError: not all arguments converted during string formatting
 [2024-07-30 12:33:22,694: WARNING/MainProcess] Call stack:
 [2024-07-30 12:33:22,694: WARNING/MainProcess]   File "/usr/local/lib/python3.10/threading.py", line 973, in _bootstrap
     self._bootstrap_inner()
 [2024-07-30 12:33:22,694: WARNING/MainProcess]   File "/usr/local/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
     self.run()
 [2024-07-30 12:33:22,694: WARNING/MainProcess]   File "/usr/local/lib/python3.10/threading.py", line 953, in run
     self._target(*self._args, **self._kwargs)
 [2024-07-30 12:33:22,694: WARNING/MainProcess]   File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 83, in _worker
     work_item.run()
 [2024-07-30 12:33:22,694: WARNING/MainProcess]   File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
     result = self.fn(*self.args, **self.kwargs)
 [2024-07-30 12:33:22,694: WARNING/MainProcess]   File "/app/api/core/indexing_runner.py", line 742, in _process_chunk
     index_processor.load(dataset, chunk_documents, with_keywords=False)
 [2024-07-30 12:33:22,694: WARNING/MainProcess]   File "/app/api/core/rag/index_processor/processor/paragraph_index_processor.py", line 60, in load
     vector.create(documents)
 [2024-07-30 12:33:22,694: WARNING/MainProcess]   File "/app/api/core/rag/datasource/vdb/vector_factory.py", line 97, in create
     embeddings = self._embeddings.embed_documents([document.page_content for document in texts])
 [2024-07-30 12:33:22,694: WARNING/MainProcess]   File "/app/api/core/embedding/cached_embedding.py", line 81, in embed_documents
     logger.error('Failed to embed documents: ', ex)
 [2024-07-30 12:33:22,694: WARNING/MainProcess] Message: 'Failed to embed documents: '
 Arguments: (InvokeBadRequestError(),)
 [2024-07-30 12:33:22,695: ERROR/MainProcess] consume document failed
 Traceback (most recent call last):
   File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 36, in invoke
     return self._invoke(model, credentials, texts, user)
   File "/app/api/core/model_runtime/model_providers/openai_api_compatible/text_embedding/text_embedding.py", line 105, in _invoke
     response.raise_for_status()  # Raise an exception for HTTP errors
   File "/app/api/.venv/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
     raise HTTPError(http_error_msg, response=self)
 requests.exceptions.HTTPError: 413 Client Error: Payload Too Large for url: http://[MASKED]/v1/embeddings
 
 During handling of the above exception, another exception occurred:
 
 Traceback (most recent call last):
   File "/app/api/core/indexing_runner.py", line 76, in run
     self._load(
   File "/app/api/core/indexing_runner.py", line 689, in _load
     tokens += future.result()
   File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 458, in result
     return self.__get_result()
   File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
     raise self._exception
   File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
     result = self.fn(*self.args, **self.kwargs)
   File "/app/api/core/indexing_runner.py", line 742, in _process_chunk
     index_processor.load(dataset, chunk_documents, with_keywords=False)
   File "/app/api/core/rag/index_processor/processor/paragraph_index_processor.py", line 60, in load
     vector.create(documents)
   File "/app/api/core/rag/datasource/vdb/vector_factory.py", line 97, in create
     embeddings = self._embeddings.embed_documents([document.page_content for document in texts])
   File "/app/api/core/embedding/cached_embedding.py", line 82, in embed_documents
     raise ex
   File "/app/api/core/embedding/cached_embedding.py", line 51, in embed_documents
     embedding_result = self._model_instance.invoke_text_embedding(
   File "/app/api/core/model_manager.py", line 170, in invoke_text_embedding
     return self._round_robin_invoke(
   File "/app/api/core/model_manager.py", line 302, in _round_robin_invoke
     return function(*args, **kwargs)
   File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 38, in invoke
     raise self._transform_invoke_error(e)
 core.model_runtime.errors.invoke.InvokeBadRequestError: [openai_api_compatible] Bad Request Error, 413 Client Error: Payload Too Large for url: http://[MASKED]/v1/embeddings
 [2024-07-30 12:33:22,699: INFO/MainProcess] Processed dataset: 698ce43b-1fd1-4334-998b-4c1bb4db92b3 latency: 0.14964738907292485
 [2024-07-30 12:33:22,724: INFO/MainProcess] Task tasks.document_indexing_task.document_indexing_task[1aebb6c4-b365-47df-9599-0b3b294742e5] succeeded in 0.17525752098299563s: None
Originally created by @mbbyn on GitHub (Jul 30, 2024). Originally assigned to: @laipz8200, @JohnJyong, @crazywoola on GitHub. ### Self Checks - [X] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [X] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [X] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [X] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:） - [X] Please do not modify this template :) and fill in all the required fields. ### Dify version 0.6.15 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce I don't know what the exact parameters are to reproduce the problem, but I can reproduce 100% of the time using this: 1. Setup an OpenAI API compatible server, such as HuggingFace TEI 2. Upload DOCX file and set chunk size to 2000 It also happens to other formats, it seems to be more likely to happen with larger chunk sizes. ### ✔️ Expected Behavior It should properly chunk and create embeddings ### ❌ Actual Behavior It fails due to sending an empty embeddings request to the embeddings server, causing the whole process to fail. ## References **NOTE**: The time difference between the logs is because of time drift on each server, they are not on the same server. ### HF TEI Logs ```json { "timestamp": "2024-07-30T12:32:16.754942Z", "level": "INFO", "message": "Success", "target": "text_embeddings_router::http::server", "filename": "router/src/http/server.rs", "line_number": 1246, "span": { "inference_time": "13.135229ms", "queue_time": "703.432µs", "tokenization_time": "627.369µs", "total_time": "14.607593ms", "name": "openai_embed" }, "spans": [ { "inference_time": "13.135229ms", "queue_time": "703.432µs", "tokenization_time": "627.369µs", "total_time": "14.607593ms", "name": "openai_embed" } ] } { "timestamp": "2024-07-30T12:32:16.763780Z", "level": "ERROR", "message": "Input validation error: `inputs` cannot be empty", "target": "text_embeddings_core::infer", "filename": "core/src/infer.rs", "line_number": 332, "span": { "normalize": true, "prompt_name": "None", "truncate": false, "truncation_direction": "Right", "name": "embed_pooled" }, "spans": [ { "name": "openai_embed" }, { "normalize": true, "prompt_name": "None", "truncate": false, "truncation_direction": "Right", "name": "embed_pooled" } ] } { "timestamp": "2024-07-30T12:33:33.908732Z", "level": "ERROR", "message": "Input validation error: `inputs` cannot be empty", "target": "text_embeddings_core::infer", "filename": "core/src/infer.rs", "line_number": 332, "span": { "normalize": true, "prompt_name": "None", "truncate": false, "truncation_direction": "Right", "name": "embed_pooled" }, "spans": [ { "name": "openai_embed" }, { "normalize": true, "prompt_name": "None", "truncate": false, "truncation_direction": "Right", "name": "embed_pooled" } ] } ``` Reproduced from the REPL ```python >>> client.embeddings.create(input="blahblabh"*100000, model="sentence-transformers/distiluse-base-multilingual-cased-v1") Traceback (most recent call last): File "<input>", line 1, in <module> client.embeddings.create(input="blahblabh"*100000, model="sentence-transformers/distiluse-base-multilingual-cased-v1") File "~/.cache/pypoetry/virtualenvs/poc-slot-machine-DH_FKYDt-py3.11/lib/python3.11/site-packages/openai/resources/embeddings.py", line 114, in create return self._post( ^^^^^^^^^^^ File "~/.cache/pypoetry/virtualenvs/poc-slot-machine-DH_FKYDt-py3.11/lib/python3.11/site-packages/openai/_base_client.py", line 1266, in post return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "~/.cache/pypoetry/virtualenvs/poc-slot-machine-DH_FKYDt-py3.11/lib/python3.11/site-packages/openai/_base_client.py", line 942, in request return self._request( ^^^^^^^^^^^^^^ File "~/.cache/pypoetry/virtualenvs/poc-slot-machine-DH_FKYDt-py3.11/lib/python3.11/site-packages/openai/_base_client.py", line 1046, in _request raise self._make_status_error_from_response(err.response) from None openai.APIStatusError: Error code: 413 - {'message': 'Input validation error: `inputs` must have less than 128000 characters. Given: 900000', 'code': 413, 'type': 'Validation'} >>> client.embeddings.create(input="", model="sentence-transformers/distiluse-base-multilingual-cased-v1") Traceback (most recent call last): File "<input>", line 1, in <module> client.embeddings.create(input="", model="sentence-transformers/distiluse-base-multilingual-cased-v1") File "~/.cache/pypoetry/virtualenvs/poc-slot-machine-DH_FKYDt-py3.11/lib/python3.11/site-packages/openai/resources/embeddings.py", line 114, in create return self._post( ^^^^^^^^^^^ File "~/.cache/pypoetry/virtualenvs/poc-slot-machine-DH_FKYDt-py3.11/lib/python3.11/site-packages/openai/_base_client.py", line 1266, in post return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "~/.cache/pypoetry/virtualenvs/poc-slot-machine-DH_FKYDt-py3.11/lib/python3.11/site-packages/openai/_base_client.py", line 942, in request return self._request( ^^^^^^^^^^^^^^ File "~/.cache/pypoetry/virtualenvs/poc-slot-machine-DH_FKYDt-py3.11/lib/python3.11/site-packages/openai/_base_client.py", line 1046, in _request raise self._make_status_error_from_response(err.response) from None openai.APIStatusError: Error code: 413 - {'message': 'Input validation error: `inputs` cannot be empty', 'code': 413, 'type': 'Validation'} ``` ### Dify Worker Logs ⚠️ Please don't pay attention to "413 Payload too large", this is not the real error. As you can see in the TEI server, the error code is indeed 413, but the actual error is that Dify is sending empty embeddings request. ```python [2024-07-30 12:32:31,515: INFO/MainProcess] Task tasks.clean_dataset_task.clean_dataset_task[a766319d-055b-4ed1-8661-a5d31cebfb1e] received [2024-07-30 12:32:31,516: INFO/MainProcess] Start clean dataset when dataset deleted: 6c33868a-5b2d-4e4f-9b5e-1deebb7ec3d8 [2024-07-30 12:32:31,520: INFO/MainProcess] Cleaning documents for dataset: 6c33868a-5b2d-4e4f-9b5e-1deebb7ec3d8 [2024-07-30 12:32:31,585: INFO/MainProcess] Cleaned dataset when dataset deleted: 6c33868a-5b2d-4e4f-9b5e-1deebb7ec3d8 latency: 0.0696456499863416 [2024-07-30 12:32:31,612: INFO/MainProcess] Task tasks.clean_dataset_task.clean_dataset_task[a766319d-055b-4ed1-8661-a5d31cebfb1e] succeeded in 0.09655854594893754s: None [2024-07-30 12:33:22,548: INFO/MainProcess] Task tasks.document_indexing_task.document_indexing_task[1aebb6c4-b365-47df-9599-0b3b294742e5] received [2024-07-30 12:33:22,552: INFO/MainProcess] Start process document: 25a7ffea-c071-47a2-b394-982ba6eb56a0 [2024-07-30 12:33:22,692: WARNING/MainProcess] --- Logging error --- [2024-07-30 12:33:22,692: WARNING/MainProcess] Traceback (most recent call last): [2024-07-30 12:33:22,692: WARNING/MainProcess] File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 36, in invoke return self._invoke(model, credentials, texts, user) [2024-07-30 12:33:22,692: WARNING/MainProcess] File "/app/api/core/model_runtime/model_providers/openai_api_compatible/text_embedding/text_embedding.py", line 105, in _invoke response.raise_for_status() # Raise an exception for HTTP errors [2024-07-30 12:33:22,692: WARNING/MainProcess] File "/app/api/.venv/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) [2024-07-30 12:33:22,693: WARNING/MainProcess] requests.exceptions.HTTPError: 413 Client Error: Payload Too Large for url: http://[MASKED]/v1/embeddings [2024-07-30 12:33:22,693: WARNING/MainProcess] During handling of the above exception, another exception occurred: [2024-07-30 12:33:22,693: WARNING/MainProcess] Traceback (most recent call last): [2024-07-30 12:33:22,693: WARNING/MainProcess] File "/app/api/core/embedding/cached_embedding.py", line 51, in embed_documents embedding_result = self._model_instance.invoke_text_embedding( [2024-07-30 12:33:22,693: WARNING/MainProcess] File "/app/api/core/model_manager.py", line 170, in invoke_text_embedding return self._round_robin_invoke( [2024-07-30 12:33:22,693: WARNING/MainProcess] File "/app/api/core/model_manager.py", line 302, in _round_robin_invoke return function(*args, **kwargs) [2024-07-30 12:33:22,693: WARNING/MainProcess] File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 38, in invoke raise self._transform_invoke_error(e) [2024-07-30 12:33:22,693: WARNING/MainProcess] core.model_runtime.errors.invoke.InvokeBadRequestError: [openai_api_compatible] Bad Request Error, 413 Client Error: Payload Too Large for url: http://[MASKED]/v1/embeddings [2024-07-30 12:33:22,693: WARNING/MainProcess] During handling of the above exception, another exception occurred: [2024-07-30 12:33:22,693: WARNING/MainProcess] Traceback (most recent call last): [2024-07-30 12:33:22,693: WARNING/MainProcess] File "/usr/local/lib/python3.10/logging/__init__.py", line 1100, in emit msg = self.format(record) [2024-07-30 12:33:22,693: WARNING/MainProcess] File "/usr/local/lib/python3.10/logging/__init__.py", line 943, in format return fmt.format(record) [2024-07-30 12:33:22,693: WARNING/MainProcess] File "/app/api/.venv/lib/python3.10/site-packages/celery/utils/log.py", line 146, in format msg = super().format(record) [2024-07-30 12:33:22,693: WARNING/MainProcess] File "/usr/local/lib/python3.10/logging/__init__.py", line 678, in format record.message = record.getMessage() [2024-07-30 12:33:22,694: WARNING/MainProcess] File "/usr/local/lib/python3.10/logging/__init__.py", line 368, in getMessage msg = msg % self.args [2024-07-30 12:33:22,694: WARNING/MainProcess] TypeError: not all arguments converted during string formatting [2024-07-30 12:33:22,694: WARNING/MainProcess] Call stack: [2024-07-30 12:33:22,694: WARNING/MainProcess] File "/usr/local/lib/python3.10/threading.py", line 973, in _bootstrap self._bootstrap_inner() [2024-07-30 12:33:22,694: WARNING/MainProcess] File "/usr/local/lib/python3.10/threading.py", line 1016, in _bootstrap_inner self.run() [2024-07-30 12:33:22,694: WARNING/MainProcess] File "/usr/local/lib/python3.10/threading.py", line 953, in run self._target(*self._args, **self._kwargs) [2024-07-30 12:33:22,694: WARNING/MainProcess] File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 83, in _worker work_item.run() [2024-07-30 12:33:22,694: WARNING/MainProcess] File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) [2024-07-30 12:33:22,694: WARNING/MainProcess] File "/app/api/core/indexing_runner.py", line 742, in _process_chunk index_processor.load(dataset, chunk_documents, with_keywords=False) [2024-07-30 12:33:22,694: WARNING/MainProcess] File "/app/api/core/rag/index_processor/processor/paragraph_index_processor.py", line 60, in load vector.create(documents) [2024-07-30 12:33:22,694: WARNING/MainProcess] File "/app/api/core/rag/datasource/vdb/vector_factory.py", line 97, in create embeddings = self._embeddings.embed_documents([document.page_content for document in texts]) [2024-07-30 12:33:22,694: WARNING/MainProcess] File "/app/api/core/embedding/cached_embedding.py", line 81, in embed_documents logger.error('Failed to embed documents: ', ex) [2024-07-30 12:33:22,694: WARNING/MainProcess] Message: 'Failed to embed documents: ' Arguments: (InvokeBadRequestError(),) [2024-07-30 12:33:22,695: ERROR/MainProcess] consume document failed Traceback (most recent call last): File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 36, in invoke return self._invoke(model, credentials, texts, user) File "/app/api/core/model_runtime/model_providers/openai_api_compatible/text_embedding/text_embedding.py", line 105, in _invoke response.raise_for_status() # Raise an exception for HTTP errors File "/app/api/.venv/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 413 Client Error: Payload Too Large for url: http://[MASKED]/v1/embeddings During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/app/api/core/indexing_runner.py", line 76, in run self._load( File "/app/api/core/indexing_runner.py", line 689, in _load tokens += future.result() File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 458, in result return self.__get_result() File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result raise self._exception File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "/app/api/core/indexing_runner.py", line 742, in _process_chunk index_processor.load(dataset, chunk_documents, with_keywords=False) File "/app/api/core/rag/index_processor/processor/paragraph_index_processor.py", line 60, in load vector.create(documents) File "/app/api/core/rag/datasource/vdb/vector_factory.py", line 97, in create embeddings = self._embeddings.embed_documents([document.page_content for document in texts]) File "/app/api/core/embedding/cached_embedding.py", line 82, in embed_documents raise ex File "/app/api/core/embedding/cached_embedding.py", line 51, in embed_documents embedding_result = self._model_instance.invoke_text_embedding( File "/app/api/core/model_manager.py", line 170, in invoke_text_embedding return self._round_robin_invoke( File "/app/api/core/model_manager.py", line 302, in _round_robin_invoke return function(*args, **kwargs) File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 38, in invoke raise self._transform_invoke_error(e) core.model_runtime.errors.invoke.InvokeBadRequestError: [openai_api_compatible] Bad Request Error, 413 Client Error: Payload Too Large for url: http://[MASKED]/v1/embeddings [2024-07-30 12:33:22,699: INFO/MainProcess] Processed dataset: 698ce43b-1fd1-4334-998b-4c1bb4db92b3 latency: 0.14964738907292485 [2024-07-30 12:33:22,724: INFO/MainProcess] Task tasks.document_indexing_task.document_indexing_task[1aebb6c4-b365-47df-9599-0b3b294742e5] succeeded in 0.17525752098299563s: None ```
yindo added the 🐞 bug label 2026-02-21 18:08:12 -05:00
yindo closed this issue 2026-02-21 18:08:12 -05:00
yindo commented 2026-02-21 18:08:12 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Jul 30, 2024):

To resolve the issue of Dify sending empty embedding requests, which leads to a 413 Payload Too Large error when uploading a DOCX file with a chunk size of 2000 in a self-hosted Docker setup, you should ensure that the file size is within the supported limit. Dify supports files under 15MB [1].

If your DOCX file exceeds this size, reduce the file size to be under 15MB. If the file size is already under 15MB and you are still encountering the issue, you may need to check the chunk size and ensure it is appropriately set to avoid sending large payloads.

Additionally, you can use the following API call to upload the document by file:

curl --location POST 'https://api.dify.ai/v1/datasets/{dataset_id}/document/create_by_file' \
--header 'Authorization: Bearer {api_key}' \
--form 'data="{
	"name": "Dify",
	"indexing_technique": "high_quality",
	"process_rule": {
		"rules": {
			"pre_processing_rules": [{
				"id": "remove_extra_spaces",
				"enabled": true
			}, {
				"id": "remove_urls_emails",
				"enabled": true
			}],
			"segmentation": {
				"separator": "###",
				"max_tokens": 500
			}
		},
		"mode": "custom"
	}
    }";
    type=text/plain' \
--form 'file=@"/path/to/file"'

Ensure that the file path and other parameters are correctly set in the API call.

For better segmentation and cleaning of the document, you can follow these steps:

  1. Segment and clean the document: Use custom segmentation rules to set an identifier such as "\n" and define the maximum segment length and overlap length. This helps in retaining semantic relevance between segments.
  2. Preprocessing rules: Replace continuous spaces, newlines, and tabs, and delete all URLs and email addresses.

By following these steps, you can ensure that the document is properly segmented and cleaned before uploading, which can help in avoiding the 413 Payload Too Large error [2][1].

To continue talking to Dosu, mention @dosu.

@dosubot[bot] commented on GitHub (Jul 30, 2024): <!-- Answer --> To resolve the issue of Dify sending empty embedding requests, which leads to a 413 Payload Too Large error when uploading a DOCX file with a chunk size of 2000 in a self-hosted Docker setup, you should ensure that the file size is within the supported limit. Dify supports files under 15MB <sup>[[1]](https://github.com/langgenius/dify-docs/blob/main/en/guides/knowledge-base/create-knowledge-and-upload-documents.md)</sup>. If your DOCX file exceeds this size, reduce the file size to be under 15MB. If the file size is already under 15MB and you are still encountering the issue, you may need to check the chunk size and ensure it is appropriately set to avoid sending large payloads. Additionally, you can use the following API call to upload the document by file: ```bash curl --location POST 'https://api.dify.ai/v1/datasets/{dataset_id}/document/create_by_file' \ --header 'Authorization: Bearer {api_key}' \ --form 'data="{ "name": "Dify", "indexing_technique": "high_quality", "process_rule": { "rules": { "pre_processing_rules": [{ "id": "remove_extra_spaces", "enabled": true }, { "id": "remove_urls_emails", "enabled": true }], "segmentation": { "separator": "###", "max_tokens": 500 } }, "mode": "custom" } }"; type=text/plain' \ --form 'file=@"/path/to/file"' ``` Ensure that the file path and other parameters are correctly set in the API call. For better segmentation and cleaning of the document, you can follow these steps: 1. **Segment and clean the document**: Use custom segmentation rules to set an identifier such as "\n" and define the maximum segment length and overlap length. This helps in retaining semantic relevance between segments. 2. **Preprocessing rules**: Replace continuous spaces, newlines, and tabs, and delete all URLs and email addresses. By following these steps, you can ensure that the document is properly segmented and cleaned before uploading, which can help in avoiding the 413 Payload Too Large error <sup>[[2]](https://github.com/langgenius/dify-docs/blob/main/en/guides/knowledge-base/maintain-dataset-via-api.md)</sup><sup>[[1]](https://github.com/langgenius/dify-docs/blob/main/en/guides/knowledge-base/create-knowledge-and-upload-documents.md)</sup>. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
yindo commented 2026-02-21 18:08:13 -05:00
Author
Owner
Copy Link
Copy Source

@JohnJyong commented on GitHub (Jul 30, 2024):

emmm，let's double comfirm, if you set chunk size as 1000 or 500 , is this issue still exist ? @mbbyn

@JohnJyong commented on GitHub (Jul 30, 2024): emmm，let's double comfirm, if you set chunk size as 1000 or 500 , is this issue still exist ? @mbbyn
yindo commented 2026-02-21 18:08:13 -05:00
Author
Owner
Copy Link
Copy Source

@mbbyn commented on GitHub (Jul 31, 2024):

@JohnJyong I spent more time trying to see where the issue is from, and I have prepared a reproduceable example as a GIF

  1. New knowledge with a simple txt file.
  2. Create everything with defaults
  3. Try updating the segment settings as seen in the GIF.
  4. Notice the error "Too Large" is simply translating the "413" http error code, but the logs from the server show it is receiving empty "inputs".

I cannot stress this enough, the error "Payload Too Large" is simply a translation for the 413 HTTP status code, but the actual error is empty inputs. For some reason, HuggingFace Text Embedding Inference uses HTTP status 413 error for empty request, when it should be 400.

dify-embedding

Dummy file:

dummy.txt

@mbbyn commented on GitHub (Jul 31, 2024): @JohnJyong I spent more time trying to see where the issue is from, and I have prepared a reproduceable example as a GIF 1. New knowledge with a simple txt file. 2. Create everything with defaults 3. Try updating the segment settings as seen in the GIF. 4. Notice the error "Too Large" is simply translating the "413" http error code, but the logs from the server show it is receiving empty "inputs". **I cannot stress this enough**, the error "Payload Too Large" is simply a translation for the 413 HTTP status code, but the actual error is empty inputs. For some reason, HuggingFace Text Embedding Inference uses HTTP status 413 error for empty request, when it should be 400. ![dify-embedding](https://github.com/user-attachments/assets/b9f73926-4e0d-4768-818d-bfecac0acd5b) Dummy file: [dummy.txt](https://github.com/user-attachments/files/16437392/dummy.txt)
yindo commented 2026-02-21 18:08:13 -05:00
Author
Owner
Copy Link
Copy Source

@mbbyn commented on GitHub (Aug 1, 2024):

I was testing another dummy data file, and this one leads to the empty request error without changing any settings at all. Simply setup the TEI server, and add this file as a knowledge with the default settings. It will fail due to empty request to TEI.

dummy.txt

@mbbyn commented on GitHub (Aug 1, 2024): I was testing another dummy data file, and this one leads to the empty request error without changing any settings at all. Simply setup the TEI server, and add this file as a knowledge with the default settings. It will fail due to empty request to TEI. [dummy.txt](https://github.com/user-attachments/files/16451587/dummy.txt)
yindo commented 2026-02-21 18:08:13 -05:00
Author
Owner
Copy Link
Copy Source

@mbbyn commented on GitHub (Aug 1, 2024):

Adding the embedding model settings, it is probably relevant for the reproduce case

image
@mbbyn commented on GitHub (Aug 1, 2024): Adding the embedding model settings, it is probably relevant for the reproduce case <img width="479" alt="image" src="https://github.com/user-attachments/assets/8af0e924-1d5e-4df1-82fb-2eba35eebdda">
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#4831
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?