langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 20:15:31 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Failed indexing csv files #8328

New Issue
Closed
opened 2026-02-21 18:25:06 -05:00 by yindo · 2 comments
yindo commented 2026-02-21 18:25:06 -05:00
Owner
Copy Link
Copy Source

Originally created by @lema-founders on GitHub (Feb 17, 2025).

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

0.15.3

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

Upload csv data using the following setting (Google text embedding and Cohere rerank model)

Image

✔️ Expected Behavior

csv files embedded successfully and files accessible within the knowledge base

❌ Actual Behavior

Failed indexing and csv files not found inside the knowledge base

Image

worker_1                 |   File "/app/api/.venv/lib/python3.12/site-packages/google/api_core/grpc_helpers.py", line 78, in error_remapped_callable
worker_1                 |     raise exceptions.from_grpc_error(exc) from exc
worker_1                 | google.api_core.exceptions.ResourceExhausted: 429 Quota exceeded for aiplatform.googleapis.com/online_prediction_requests_per_base_model with base model: textembedding-gecko. Please submit a quota increase request. https://cloud.google.com/vertex-ai/docs/generative-ai/quotas-genai.
worker_1                 | 
worker_1                 | During handling of the above exception, another exception occurred:
worker_1                 | 
worker_1                 | Traceback (most recent call last):
worker_1                 |   File "/app/api/core/rag/embedding/cached_embedding.py", line 61, in embed_documents
worker_1                 |     embedding_result = self._model_instance.invoke_text_embedding(
worker_1                 |                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/core/model_manager.py", line 185, in invoke_text_embedding
worker_1                 |     self._round_robin_invoke(
worker_1                 |   File "/app/api/core/model_manager.py", line 334, in _round_robin_invoke
worker_1                 |     return function(*args, **kwargs)
worker_1                 |            ^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 46, in invoke
worker_1                 |     raise self._transform_invoke_error(e)
worker_1                 |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/core/model_runtime/model_providers/__base/ai_model.py", line 70, in _transform_invoke_error
worker_1                 |     for invoke_error, model_errors in self._invoke_error_mapping.items():
worker_1                 |                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/core/model_runtime/model_providers/vertex_ai/_common.py", line 15, in _invoke_error_mapping
worker_1                 |     raise NotImplementedError
worker_1                 | NotImplementedError
worker_1                 | 2025-02-17 07:44:15.616 ERROR [ThreadPoolExecutor-17_9] [cached_embedding.py:98] - Failed to embed documents: %s
worker_1                 | Traceback (most recent call last):
worker_1                 |   File "/app/api/.venv/lib/python3.12/site-packages/google/api_core/grpc_helpers.py", line 76, in error_remapped_callable
worker_1                 |     return callable_(*args, **kwargs)
worker_1                 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/.venv/lib/python3.12/site-packages/grpc/_channel.py", line 1181, in __call__
worker_1                 |     return _end_unary_response_blocking(state, call, False, None)
worker_1                 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/.venv/lib/python3.12/site-packages/grpc/_channel.py", line 1006, in _end_unary_response_blocking
worker_1                 |     raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
worker_1                 |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 | grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
worker_1                 |      status = StatusCode.RESOURCE_EXHAUSTED
worker_1                 |      details = "Quota exceeded for aiplatform.googleapis.com/online_prediction_requests_per_base_model with base model: textembedding-gecko. Please submit a quota increase request. https://cloud.google.com/vertex-ai/docs/generative-ai/quotas-genai."
worker_1                 |      debug_error_string = "UNKNOWN:Error received from peer ipv4:108.177.98.95:443 {created_time:"2025-02-17T07:44:15.614797592+00:00", grpc_status:8, grpc_message:"Quota exceeded for aiplatform.googleapis.com/online_prediction_requests_per_base_model with base model: textembedding-gecko. Please submit a quota increase request. https://cloud.google.com/vertex-ai/docs/generative-ai/quotas-genai."}"
worker_1                 | >
worker_1                 | 
worker_1                 | The above exception was the direct cause of the following exception:
worker_1                 | 
worker_1                 | Traceback (most recent call last):
worker_1                 |   File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 44, in invoke
worker_1                 |     return self._invoke(model, credentials, texts, user, input_type)
worker_1                 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/core/model_runtime/model_providers/vertex_ai/text_embedding/text_embedding.py", line 69, in _invoke
worker_1                 |     embeddings_batch, embedding_used_tokens = self._embedding_invoke(client=client, texts=texts)
worker_1                 |                                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/core/model_runtime/model_providers/vertex_ai/text_embedding/text_embedding.py", line 140, in _embedding_invoke
worker_1                 |     response = client.get_embeddings(texts)
worker_1                 |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/.venv/lib/python3.12/site-packages/vertexai/language_models/_language_models.py", line 2097, in get_embeddings
worker_1                 |     prediction_response = self._endpoint.predict(
worker_1                 |                           ^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/.venv/lib/python3.12/site-packages/google/cloud/aiplatform/models.py", line 1597, in predict
worker_1                 |     prediction_response = self._prediction_client.predict(
worker_1                 |                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/.venv/lib/python3.12/site-packages/google/cloud/aiplatform_v1/services/prediction_service/client.py", line 836, in predict
worker_1                 |     response = rpc(
worker_1                 |                ^^^^
worker_1                 |   File "/app/api/.venv/lib/python3.12/site-packages/google/api_core/gapic_v1/method.py", line 131, in __call__
worker_1                 |     return wrapped_func(*args, **kwargs)
worker_1                 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/.venv/lib/python3.12/site-packages/google/api_core/grpc_helpers.py", line 78, in error_remapped_callable
worker_1                 |     raise exceptions.from_grpc_error(exc) from exc
worker_1                 | google.api_core.exceptions.ResourceExhausted: 429 Quota exceeded for aiplatform.googleapis.com/online_prediction_requests_per_base_model with base model: textembedding-gecko. Please submit a quota increase request. https://cloud.google.com/vertex-ai/docs/generative-ai/quotas-genai.
worker_1                 | 
worker_1                 | During handling of the above exception, another exception occurred:
worker_1                 | 
worker_1                 | Traceback (most recent call last):
worker_1                 |   File "/app/api/core/rag/embedding/cached_embedding.py", line 61, in embed_documents
worker_1                 |     embedding_result = self._model_instance.invoke_text_embedding(
worker_1                 |                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/core/model_manager.py", line 185, in invoke_text_embedding
worker_1                 |     self._round_robin_invoke(
worker_1                 |   File "/app/api/core/model_manager.py", line 334, in _round_robin_invoke
worker_1                 |     return function(*args, **kwargs)
worker_1                 |            ^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 46, in invoke
worker_1                 |     raise self._transform_invoke_error(e)
worker_1                 |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/core/model_runtime/model_providers/__base/ai_model.py", line 70, in _transform_invoke_error
worker_1                 |     for invoke_error, model_errors in self._invoke_error_mapping.items():
worker_1                 |                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/core/model_runtime/model_providers/vertex_ai/_common.py", line 15, in _invoke_error_mapping
worker_1                 |     raise NotImplementedError
worker_1                 | NotImplementedError
worker_1                 | 2025-02-17 07:44:15.620 ERROR [ThreadPoolExecutor-17_6] [cached_embedding.py:98] - Failed to embed documents: %s
worker_1                 | Traceback (most recent call last):
worker_1                 |   File "/app/api/.venv/lib/python3.12/site-packages/google/api_core/grpc_helpers.py", line 76, in error_remapped_callable
worker_1                 |     return callable_(*args, **kwargs)
worker_1                 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/.venv/lib/python3.12/site-packages/grpc/_channel.py", line 1181, in __call__
worker_1                 |     return _end_unary_response_blocking(state, call, False, None)
worker_1                 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/.venv/lib/python3.12/site-packages/grpc/_channel.py", line 1006, in _end_unary_response_blocking
worker_1                 |     raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
worker_1                 |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 | grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
worker_1                 |      status = StatusCode.RESOURCE_EXHAUSTED
worker_1                 |      details = "Quota exceeded for aiplatform.googleapis.com/online_prediction_requests_per_base_model with base model: textembedding-gecko. Please submit a quota increase request. https://cloud.google.com/vertex-ai/docs/generative-ai/quotas-genai."
worker_1                 |      debug_error_string = "UNKNOWN:Error received from peer ipv4:142.251.188.95:443 {created_time:"2025-02-17T07:44:15.615561045+00:00", grpc_status:8, grpc_message:"Quota exceeded for aiplatform.googleapis.com/online_prediction_requests_per_base_model with base model: textembedding-gecko. Please submit a quota increase request. https://cloud.google.com/vertex-ai/docs/generative-ai/quotas-genai."}"
worker_1                 | >
worker_1                 | 
worker_1                 | The above exception was the direct cause of the following exception:
worker_1                 | 
worker_1                 | Traceback (most recent call last):
worker_1                 |   File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 44, in invoke
worker_1                 |     return self._invoke(model, credentials, texts, user, input_type)
worker_1                 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/core/model_runtime/model_providers/vertex_ai/text_embedding/text_embedding.py", line 69, in _invoke
worker_1                 |     embeddings_batch, embedding_used_tokens = self._embedding_invoke(client=client, texts=texts)
worker_1                 |                                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/core/model_runtime/model_providers/vertex_ai/text_embedding/text_embedding.py", line 140, in _embedding_invoke
worker_1                 |     response = client.get_embeddings(texts)
worker_1                 |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/.venv/lib/python3.12/site-packages/vertexai/language_models/_language_models.py", line 2097, in get_embeddings
worker_1                 |     prediction_response = self._endpoint.predict(
worker_1                 |                           ^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/.venv/lib/python3.12/site-packages/google/cloud/aiplatform/models.py", line 1597, in predict
worker_1                 |     prediction_response = self._prediction_client.predict(
worker_1                 |                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/.venv/lib/python3.12/site-packages/google/cloud/aiplatform_v1/services/prediction_service/client.py", line 836, in predict
worker_1                 |     response = rpc(
worker_1                 |                ^^^^
worker_1                 |   File "/app/api/.venv/lib/python3.12/site-packages/google/api_core/gapic_v1/method.py", line 131, in __call__
worker_1                 |     return wrapped_func(*args, **kwargs)
worker_1                 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/.venv/lib/python3.12/site-packages/google/api_core/grpc_helpers.py", line 78, in error_remapped_callable
worker_1                 |     raise exceptions.from_grpc_error(exc) from exc
worker_1                 | google.api_core.exceptions.ResourceExhausted: 429 Quota exceeded for aiplatform.googleapis.com/online_prediction_requests_per_base_model with base model: textembedding-gecko. Please submit a quota increase request. https://cloud.google.com/vertex-ai/docs/generative-ai/quotas-genai.
worker_1                 | 
worker_1                 | During handling of the above exception, another exception occurred:
worker_1                 | 
worker_1                 | Traceback (most recent call last):
worker_1                 |   File "/app/api/core/rag/embedding/cached_embedding.py", line 61, in embed_documents
worker_1                 |     embedding_result = self._model_instance.invoke_text_embedding(
worker_1                 |                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/core/model_manager.py", line 185, in invoke_text_embedding
worker_1                 |     self._round_robin_invoke(
worker_1                 |   File "/app/api/core/model_manager.py", line 334, in _round_robin_invoke
worker_1                 |     return function(*args, **kwargs)
worker_1                 |            ^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 46, in invoke
worker_1                 |     raise self._transform_invoke_error(e)
worker_1                 |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/core/model_runtime/model_providers/__base/ai_model.py", line 70, in _transform_invoke_error
worker_1                 |     for invoke_error, model_errors in self._invoke_error_mapping.items():
worker_1                 |                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/core/model_runtime/model_providers/vertex_ai/_common.py", line 15, in _invoke_error_mapping
worker_1                 |     raise NotImplementedError
worker_1                 | NotImplementedError
worker_1                 | 2025-02-17 07:44:15.623 ERROR [ThreadPoolExecutor-17_0] [cached_embedding.py:98] - Failed to embed documents: %s
worker_1                 | Traceback (most recent call last):
worker_1                 |   File "/app/api/.venv/lib/python3.12/site-packages/google/api_core/grpc_helpers.py", line 76, in error_remapped_callable
worker_1                 |     return callable_(*args, **kwargs)
worker_1                 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/.venv/lib/python3.12/site-packages/grpc/_channel.py", line 1181, in __call__
worker_1                 |     return _end_unary_response_blocking(state, call, False, None)
worker_1                 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/.venv/lib/python3.12/site-packages/grpc/_channel.py", line 1006, in _end_unary_response_blocking
worker_1                 |     raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
worker_1                 |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 | grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
worker_1                 |      status = StatusCode.RESOURCE_EXHAUSTED
worker_1                 |      details = "Quota exceeded for aiplatform.googleapis.com/online_prediction_requests_per_base_model with base model: textembedding-gecko. Please submit a quota increase request. https://cloud.google.com/vertex-ai/docs/generative-ai/quotas-genai."
worker_1                 |      debug_error_string = "UNKNOWN:Error received from peer ipv4:74.125.135.95:443 {created_time:"2025-02-17T07:44:15.617492549+00:00", grpc_status:8, grpc_message:"Quota exceeded for aiplatform.googleapis.com/online_prediction_requests_per_base_model with base model: textembedding-gecko. Please submit a quota increase request. https://cloud.google.com/vertex-ai/docs/generative-ai/quotas-genai."}"
worker_1                 | >
worker_1                 | 
worker_1                 | The above exception was the direct cause of the following exception:
worker_1                 | 
worker_1                 | Traceback (most recent call last):
worker_1                 |   File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 44, in invoke
worker_1                 |     return self._invoke(model, credentials, texts, user, input_type)
worker_1                 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/core/model_runtime/model_providers/vertex_ai/text_embedding/text_embedding.py", line 69, in _invoke
worker_1                 |     embeddings_batch, embedding_used_tokens = self._embedding_invoke(client=client, texts=texts)
worker_1                 |                                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/core/model_runtime/model_providers/vertex_ai/text_embedding/text_embedding.py", line 140, in _embedding_invoke
worker_1                 |     response = client.get_embeddings(texts)
worker_1                 |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/.venv/lib/python3.12/site-packages/vertexai/language_models/_language_models.py", line 2097, in get_embeddings
worker_1                 |     prediction_response = self._endpoint.predict(
worker_1                 |                           ^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/.venv/lib/python3.12/site-packages/google/cloud/aiplatform/models.py", line 1597, in predict
worker_1                 |     prediction_response = self._prediction_client.predict(
worker_1                 |                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/.venv/lib/python3.12/site-packages/google/cloud/aiplatform_v1/services/prediction_service/client.py", line 836, in predict
worker_1                 |     response = rpc(
worker_1                 |                ^^^^
worker_1                 |   File "/app/api/.venv/lib/python3.12/site-packages/google/api_core/gapic_v1/method.py", line 131, in __call__
worker_1                 |     return wrapped_func(*args, **kwargs)
worker_1                 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/.venv/lib/python3.12/site-packages/google/api_core/grpc_helpers.py", line 78, in error_remapped_callable
worker_1                 |     raise exceptions.from_grpc_error(exc) from exc
worker_1                 | google.api_core.exceptions.ResourceExhausted: 429 Quota exceeded for aiplatform.googleapis.com/online_prediction_requests_per_base_model with base model: textembedding-gecko. Please submit a quota increase request. https://cloud.google.com/vertex-ai/docs/generative-ai/quotas-genai.
worker_1                 | 
worker_1                 | During handling of the above exception, another exception occurred:
worker_1                 | 
worker_1                 | Traceback (most recent call last):
worker_1                 |   File "/app/api/core/rag/embedding/cached_embedding.py", line 61, in embed_documents
worker_1                 |     embedding_result = self._model_instance.invoke_text_embedding(
worker_1                 |                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/core/model_manager.py", line 185, in invoke_text_embedding
worker_1                 |     self._round_robin_invoke(
worker_1                 |   File "/app/api/core/model_manager.py", line 334, in _round_robin_invoke
worker_1                 |     return function(*args, **kwargs)
worker_1                 |            ^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 46, in invoke
worker_1                 |     raise self._transform_invoke_error(e)
worker_1                 |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/core/model_runtime/model_providers/__base/ai_model.py", line 70, in _transform_invoke_error
worker_1                 |     for invoke_error, model_errors in self._invoke_error_mapping.items():
worker_1                 |                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/core/model_runtime/model_providers/vertex_ai/_common.py", line 15, in _invoke_error_mapping
worker_1                 |     raise NotImplementedError
worker_1                 | NotImplementedError
worker_1                 | 2025-02-17 07:44:15.870 ERROR [ThreadPoolExecutor-17_7] [cached_embedding.py:98] - Failed to embed documents: %s
worker_1                 | Traceback (most recent call last):
worker_1                 |   File "/app/api/.venv/lib/python3.12/site-packages/google/api_core/grpc_helpers.py", line 76, in error_remapped_callable
worker_1                 |     return callable_(*args, **kwargs)
worker_1                 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/.venv/lib/python3.12/site-packages/grpc/_channel.py", line 1181, in __call__
worker_1                 |     return _end_unary_response_blocking(state, call, False, None)
worker_1                 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/.venv/lib/python3.12/site-packages/grpc/_channel.py", line 1006, in _end_unary_response_blocking
worker_1                 |     raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
worker_1                 |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 | grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
worker_1                 |      status = StatusCode.RESOURCE_EXHAUSTED
worker_1                 |      details = "Quota exceeded for aiplatform.googleapis.com/online_prediction_requests_per_base_model with base model: textembedding-gecko. Please submit a quota increase request. https://cloud.google.com/vertex-ai/docs/generative-ai/quotas-genai."
worker_1                 |      debug_error_string = "UNKNOWN:Error received from peer ipv4:74.125.197.95:443 {created_time:"2025-02-17T07:44:15.869419385+00:00", grpc_status:8, grpc_message:"Quota exceeded for aiplatform.googleapis.com/online_prediction_requests_per_base_model with base model: textembedding-gecko. Please submit a quota increase request. https://cloud.google.com/vertex-ai/docs/generative-ai/quotas-genai."}"
worker_1                 | >
worker_1                 | 
worker_1                 | The above exception was the direct cause of the following exception:
worker_1                 | 
worker_1                 | Traceback (most recent call last):
worker_1                 |   File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 44, in invoke
worker_1                 |     return self._invoke(model, credentials, texts, user, input_type)
worker_1                 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/core/model_runtime/model_providers/vertex_ai/text_embedding/text_embedding.py", line 69, in _invoke
worker_1                 |     embeddings_batch, embedding_used_tokens = self._embedding_invoke(client=client, texts=texts)
worker_1                 |                                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/core/model_runtime/model_providers/vertex_ai/text_embedding/text_embedding.py", line 140, in _embedding_invoke
worker_1                 |     response = client.get_embeddings(texts)
worker_1                 |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/.venv/lib/python3.12/site-packages/vertexai/language_models/_language_models.py", line 2097, in get_embeddings
worker_1                 |     prediction_response = self._endpoint.predict(
worker_1                 |                           ^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/.venv/lib/python3.12/site-packages/google/cloud/aiplatform/models.py", line 1597, in predict
worker_1                 |     prediction_response = self._prediction_client.predict(
worker_1                 |                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/.venv/lib/python3.12/site-packages/google/cloud/aiplatform_v1/services/prediction_service/client.py", line 836, in predict
worker_1                 |     response = rpc(
worker_1                 |                ^^^^
worker_1                 |   File "/app/api/.venv/lib/python3.12/site-packages/google/api_core/gapic_v1/method.py", line 131, in __call__
worker_1                 |     return wrapped_func(*args, **kwargs)
worker_1                 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/.venv/lib/python3.12/site-packages/google/api_core/grpc_helpers.py", line 78, in error_remapped_callable
worker_1                 |     raise exceptions.from_grpc_error(exc) from exc
worker_1                 | google.api_core.exceptions.ResourceExhausted: 429 Quota exceeded for aiplatform.googleapis.com/online_prediction_requests_per_base_model with base model: textembedding-gecko. Please submit a quota increase request. https://cloud.google.com/vertex-ai/docs/generative-ai/quotas-genai.
worker_1                 | 
worker_1                 | During handling of the above exception, another exception occurred:
worker_1                 | 
worker_1                 | Traceback (most recent call last):
worker_1                 |   File "/app/api/core/rag/embedding/cached_embedding.py", line 61, in embed_documents
worker_1                 |     embedding_result = self._model_instance.invoke_text_embedding(
worker_1                 |                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/core/model_manager.py", line 185, in invoke_text_embedding
worker_1                 |     self._round_robin_invoke(
worker_1                 |   File "/app/api/core/model_manager.py", line 334, in _round_robin_invoke
worker_1                 |     return function(*args, **kwargs)
worker_1                 |            ^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 46, in invoke
worker_1                 |     raise self._transform_invoke_error(e)
worker_1                 |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/core/model_runtime/model_providers/__base/ai_model.py", line 70, in _transform_invoke_error
worker_1                 |     for invoke_error, model_errors in self._invoke_error_mapping.items():
worker_1                 |                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/core/model_runtime/model_providers/vertex_ai/_common.py", line 15, in _invoke_error_mapping
worker_1                 |     raise NotImplementedError
worker_1                 | NotImplementedError
worker_1                 | 2025-02-17 07:44:15.873 ERROR [Dummy-5] [indexing_runner.py:96] - consume document failed
worker_1                 | Traceback (most recent call last):
worker_1                 |   File "/app/api/.venv/lib/python3.12/site-packages/google/api_core/grpc_helpers.py", line 76, in error_remapped_callable
worker_1                 |     return callable_(*args, **kwargs)
worker_1                 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/.venv/lib/python3.12/site-packages/grpc/_channel.py", line 1181, in __call__
worker_1                 |     return _end_unary_response_blocking(state, call, False, None)
worker_1                 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/.venv/lib/python3.12/site-packages/grpc/_channel.py", line 1006, in _end_unary_response_blocking
worker_1                 |     raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
worker_1                 |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 | grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
worker_1                 |      status = StatusCode.RESOURCE_EXHAUSTED
worker_1                 |      details = "Quota exceeded for aiplatform.googleapis.com/online_prediction_requests_per_base_model with base model: textembedding-gecko. Please submit a quota increase request. https://cloud.google.com/vertex-ai/docs/generative-ai/quotas-genai."
worker_1                 |      debug_error_string = "UNKNOWN:Error received from peer ipv4:74.125.135.95:443 {created_time:"2025-02-17T07:44:15.617492549+00:00", grpc_status:8, grpc_message:"Quota exceeded for aiplatform.googleapis.com/online_prediction_requests_per_base_model with base model: textembedding-gecko. Please submit a quota increase request. https://cloud.google.com/vertex-ai/docs/generative-ai/quotas-genai."}"
worker_1                 | >
worker_1                 | 
worker_1                 | The above exception was the direct cause of the following exception:
worker_1                 | 
worker_1                 | Traceback (most recent call last):
worker_1                 |   File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 44, in invoke
worker_1                 |     return self._invoke(model, credentials, texts, user, input_type)
worker_1                 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/core/model_runtime/model_providers/vertex_ai/text_embedding/text_embedding.py", line 69, in _invoke
worker_1                 |     embeddings_batch, embedding_used_tokens = self._embedding_invoke(client=client, texts=texts)
worker_1                 |                                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/core/model_runtime/model_providers/vertex_ai/text_embedding/text_embedding.py", line 140, in _embedding_invoke
worker_1                 |     response = client.get_embeddings(texts)
worker_1                 |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/.venv/lib/python3.12/site-packages/vertexai/language_models/_language_models.py", line 2097, in get_embeddings
worker_1                 |     prediction_response = self._endpoint.predict(
worker_1                 |                           ^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/.venv/lib/python3.12/site-packages/google/cloud/aiplatform/models.py", line 1597, in predict
worker_1                 |     prediction_response = self._prediction_client.predict(
worker_1                 |                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/.venv/lib/python3.12/site-packages/google/cloud/aiplatform_v1/services/prediction_service/client.py", line 836, in predict
worker_1                 |     response = rpc(
worker_1                 |                ^^^^
worker_1                 |   File "/app/api/.venv/lib/python3.12/site-packages/google/api_core/gapic_v1/method.py", line 131, in __call__
worker_1                 |     return wrapped_func(*args, **kwargs)
worker_1                 |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/.venv/lib/python3.12/site-packages/google/api_core/grpc_helpers.py", line 78, in error_remapped_callable
worker_1                 |     raise exceptions.from_grpc_error(exc) from exc
worker_1                 | google.api_core.exceptions.ResourceExhausted: 429 Quota exceeded for aiplatform.googleapis.com/online_prediction_requests_per_base_model with base model: textembedding-gecko. Please submit a quota increase request. https://cloud.google.com/vertex-ai/docs/generative-ai/quotas-genai.
worker_1                 | 
worker_1                 | During handling of the above exception, another exception occurred:
worker_1                 | 
worker_1                 | Traceback (most recent call last):
worker_1                 |   File "/app/api/core/indexing_runner.py", line 80, in run
worker_1                 |     self._load(
worker_1                 |   File "/app/api/core/indexing_runner.py", line 570, in _load
worker_1                 |     tokens += future.result()
worker_1                 |               ^^^^^^^^^^^^^^^
worker_1                 |   File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 456, in result
worker_1                 |     return self.__get_result()
worker_1                 |            ^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
worker_1                 |     raise self._exception
worker_1                 |   File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 59, in run
worker_1                 |     result = self.fn(*self.args, **self.kwargs)
worker_1                 |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/core/indexing_runner.py", line 627, in _process_chunk
worker_1                 |     index_processor.load(dataset, chunk_documents, with_keywords=False)
worker_1                 |   File "/app/api/core/rag/index_processor/processor/paragraph_index_processor.py", line 78, in load
worker_1                 |     vector.create(documents)
worker_1                 |   File "/app/api/core/rag/datasource/vdb/vector_factory.py", line 156, in create
worker_1                 |     embeddings = self._embeddings.embed_documents([document.page_content for document in texts])
worker_1                 |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/core/rag/embedding/cached_embedding.py", line 99, in embed_documents
worker_1                 |     raise ex
worker_1                 |   File "/app/api/core/rag/embedding/cached_embedding.py", line 61, in embed_documents
worker_1                 |     embedding_result = self._model_instance.invoke_text_embedding(
worker_1                 |                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/core/model_manager.py", line 185, in invoke_text_embedding
worker_1                 |     self._round_robin_invoke(
worker_1                 |   File "/app/api/core/model_manager.py", line 334, in _round_robin_invoke
worker_1                 |     return function(*args, **kwargs)
worker_1                 |            ^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 46, in invoke
worker_1                 |     raise self._transform_invoke_error(e)
worker_1                 |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/core/model_runtime/model_providers/__base/ai_model.py", line 70, in _transform_invoke_error
worker_1                 |     for invoke_error, model_errors in self._invoke_error_mapping.items():
worker_1                 |                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^
worker_1                 |   File "/app/api/core/model_runtime/model_providers/vertex_ai/_common.py", line 15, in _invoke_error_mapping
worker_1                 |     raise NotImplementedError
worker_1                 | NotImplementedError
worker_1                 | 2025-02-17 07:44:15.881 INFO [Dummy-5] [retry_document_indexing_task.py:96] - Retry dataset: 0191910d-eced-4e73-b42b-d02bbbc2ed81 latency: 9.137617005966604
worker_1                 | 2025-02-17 07:44:15.929 INFO [Dummy-5] [trace.py:128] - Task tasks.retry_document_indexing_task.retry_document_indexing_task[aeb5d7b7-3123-47a4-8026-4a5f04a35026] succeeded in 9.186373426113278s: None
Originally created by @lema-founders on GitHub (Feb 17, 2025). ### Self Checks - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:） - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version 0.15.3 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce Upload csv data using the following setting (Google text embedding and Cohere rerank model) ![Image](https://github.com/user-attachments/assets/c5f1a131-7418-40e5-8fde-cf9404f99daf) ### ✔️ Expected Behavior csv files embedded successfully and files accessible within the knowledge base ### ❌ Actual Behavior Failed indexing and csv files not found inside the knowledge base ![Image](https://github.com/user-attachments/assets/c5f1a131-7418-40e5-8fde-cf9404f99daf) ``` worker_1 | File "/app/api/.venv/lib/python3.12/site-packages/google/api_core/grpc_helpers.py", line 78, in error_remapped_callable worker_1 | raise exceptions.from_grpc_error(exc) from exc worker_1 | google.api_core.exceptions.ResourceExhausted: 429 Quota exceeded for aiplatform.googleapis.com/online_prediction_requests_per_base_model with base model: textembedding-gecko. Please submit a quota increase request. https://cloud.google.com/vertex-ai/docs/generative-ai/quotas-genai. worker_1 | worker_1 | During handling of the above exception, another exception occurred: worker_1 | worker_1 | Traceback (most recent call last): worker_1 | File "/app/api/core/rag/embedding/cached_embedding.py", line 61, in embed_documents worker_1 | embedding_result = self._model_instance.invoke_text_embedding( worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/core/model_manager.py", line 185, in invoke_text_embedding worker_1 | self._round_robin_invoke( worker_1 | File "/app/api/core/model_manager.py", line 334, in _round_robin_invoke worker_1 | return function(*args, **kwargs) worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 46, in invoke worker_1 | raise self._transform_invoke_error(e) worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/core/model_runtime/model_providers/__base/ai_model.py", line 70, in _transform_invoke_error worker_1 | for invoke_error, model_errors in self._invoke_error_mapping.items(): worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/core/model_runtime/model_providers/vertex_ai/_common.py", line 15, in _invoke_error_mapping worker_1 | raise NotImplementedError worker_1 | NotImplementedError worker_1 | 2025-02-17 07:44:15.616 ERROR [ThreadPoolExecutor-17_9] [cached_embedding.py:98] - Failed to embed documents: %s worker_1 | Traceback (most recent call last): worker_1 | File "/app/api/.venv/lib/python3.12/site-packages/google/api_core/grpc_helpers.py", line 76, in error_remapped_callable worker_1 | return callable_(*args, **kwargs) worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/.venv/lib/python3.12/site-packages/grpc/_channel.py", line 1181, in __call__ worker_1 | return _end_unary_response_blocking(state, call, False, None) worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/.venv/lib/python3.12/site-packages/grpc/_channel.py", line 1006, in _end_unary_response_blocking worker_1 | raise _InactiveRpcError(state) # pytype: disable=not-instantiable worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: worker_1 | status = StatusCode.RESOURCE_EXHAUSTED worker_1 | details = "Quota exceeded for aiplatform.googleapis.com/online_prediction_requests_per_base_model with base model: textembedding-gecko. Please submit a quota increase request. https://cloud.google.com/vertex-ai/docs/generative-ai/quotas-genai." worker_1 | debug_error_string = "UNKNOWN:Error received from peer ipv4:108.177.98.95:443 {created_time:"2025-02-17T07:44:15.614797592+00:00", grpc_status:8, grpc_message:"Quota exceeded for aiplatform.googleapis.com/online_prediction_requests_per_base_model with base model: textembedding-gecko. Please submit a quota increase request. https://cloud.google.com/vertex-ai/docs/generative-ai/quotas-genai."}" worker_1 | > worker_1 | worker_1 | The above exception was the direct cause of the following exception: worker_1 | worker_1 | Traceback (most recent call last): worker_1 | File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 44, in invoke worker_1 | return self._invoke(model, credentials, texts, user, input_type) worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/core/model_runtime/model_providers/vertex_ai/text_embedding/text_embedding.py", line 69, in _invoke worker_1 | embeddings_batch, embedding_used_tokens = self._embedding_invoke(client=client, texts=texts) worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/core/model_runtime/model_providers/vertex_ai/text_embedding/text_embedding.py", line 140, in _embedding_invoke worker_1 | response = client.get_embeddings(texts) worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/.venv/lib/python3.12/site-packages/vertexai/language_models/_language_models.py", line 2097, in get_embeddings worker_1 | prediction_response = self._endpoint.predict( worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/.venv/lib/python3.12/site-packages/google/cloud/aiplatform/models.py", line 1597, in predict worker_1 | prediction_response = self._prediction_client.predict( worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/.venv/lib/python3.12/site-packages/google/cloud/aiplatform_v1/services/prediction_service/client.py", line 836, in predict worker_1 | response = rpc( worker_1 | ^^^^ worker_1 | File "/app/api/.venv/lib/python3.12/site-packages/google/api_core/gapic_v1/method.py", line 131, in __call__ worker_1 | return wrapped_func(*args, **kwargs) worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/.venv/lib/python3.12/site-packages/google/api_core/grpc_helpers.py", line 78, in error_remapped_callable worker_1 | raise exceptions.from_grpc_error(exc) from exc worker_1 | google.api_core.exceptions.ResourceExhausted: 429 Quota exceeded for aiplatform.googleapis.com/online_prediction_requests_per_base_model with base model: textembedding-gecko. Please submit a quota increase request. https://cloud.google.com/vertex-ai/docs/generative-ai/quotas-genai. worker_1 | worker_1 | During handling of the above exception, another exception occurred: worker_1 | worker_1 | Traceback (most recent call last): worker_1 | File "/app/api/core/rag/embedding/cached_embedding.py", line 61, in embed_documents worker_1 | embedding_result = self._model_instance.invoke_text_embedding( worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/core/model_manager.py", line 185, in invoke_text_embedding worker_1 | self._round_robin_invoke( worker_1 | File "/app/api/core/model_manager.py", line 334, in _round_robin_invoke worker_1 | return function(*args, **kwargs) worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 46, in invoke worker_1 | raise self._transform_invoke_error(e) worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/core/model_runtime/model_providers/__base/ai_model.py", line 70, in _transform_invoke_error worker_1 | for invoke_error, model_errors in self._invoke_error_mapping.items(): worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/core/model_runtime/model_providers/vertex_ai/_common.py", line 15, in _invoke_error_mapping worker_1 | raise NotImplementedError worker_1 | NotImplementedError worker_1 | 2025-02-17 07:44:15.620 ERROR [ThreadPoolExecutor-17_6] [cached_embedding.py:98] - Failed to embed documents: %s worker_1 | Traceback (most recent call last): worker_1 | File "/app/api/.venv/lib/python3.12/site-packages/google/api_core/grpc_helpers.py", line 76, in error_remapped_callable worker_1 | return callable_(*args, **kwargs) worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/.venv/lib/python3.12/site-packages/grpc/_channel.py", line 1181, in __call__ worker_1 | return _end_unary_response_blocking(state, call, False, None) worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/.venv/lib/python3.12/site-packages/grpc/_channel.py", line 1006, in _end_unary_response_blocking worker_1 | raise _InactiveRpcError(state) # pytype: disable=not-instantiable worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: worker_1 | status = StatusCode.RESOURCE_EXHAUSTED worker_1 | details = "Quota exceeded for aiplatform.googleapis.com/online_prediction_requests_per_base_model with base model: textembedding-gecko. Please submit a quota increase request. https://cloud.google.com/vertex-ai/docs/generative-ai/quotas-genai." worker_1 | debug_error_string = "UNKNOWN:Error received from peer ipv4:142.251.188.95:443 {created_time:"2025-02-17T07:44:15.615561045+00:00", grpc_status:8, grpc_message:"Quota exceeded for aiplatform.googleapis.com/online_prediction_requests_per_base_model with base model: textembedding-gecko. Please submit a quota increase request. https://cloud.google.com/vertex-ai/docs/generative-ai/quotas-genai."}" worker_1 | > worker_1 | worker_1 | The above exception was the direct cause of the following exception: worker_1 | worker_1 | Traceback (most recent call last): worker_1 | File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 44, in invoke worker_1 | return self._invoke(model, credentials, texts, user, input_type) worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/core/model_runtime/model_providers/vertex_ai/text_embedding/text_embedding.py", line 69, in _invoke worker_1 | embeddings_batch, embedding_used_tokens = self._embedding_invoke(client=client, texts=texts) worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/core/model_runtime/model_providers/vertex_ai/text_embedding/text_embedding.py", line 140, in _embedding_invoke worker_1 | response = client.get_embeddings(texts) worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/.venv/lib/python3.12/site-packages/vertexai/language_models/_language_models.py", line 2097, in get_embeddings worker_1 | prediction_response = self._endpoint.predict( worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/.venv/lib/python3.12/site-packages/google/cloud/aiplatform/models.py", line 1597, in predict worker_1 | prediction_response = self._prediction_client.predict( worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/.venv/lib/python3.12/site-packages/google/cloud/aiplatform_v1/services/prediction_service/client.py", line 836, in predict worker_1 | response = rpc( worker_1 | ^^^^ worker_1 | File "/app/api/.venv/lib/python3.12/site-packages/google/api_core/gapic_v1/method.py", line 131, in __call__ worker_1 | return wrapped_func(*args, **kwargs) worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/.venv/lib/python3.12/site-packages/google/api_core/grpc_helpers.py", line 78, in error_remapped_callable worker_1 | raise exceptions.from_grpc_error(exc) from exc worker_1 | google.api_core.exceptions.ResourceExhausted: 429 Quota exceeded for aiplatform.googleapis.com/online_prediction_requests_per_base_model with base model: textembedding-gecko. Please submit a quota increase request. https://cloud.google.com/vertex-ai/docs/generative-ai/quotas-genai. worker_1 | worker_1 | During handling of the above exception, another exception occurred: worker_1 | worker_1 | Traceback (most recent call last): worker_1 | File "/app/api/core/rag/embedding/cached_embedding.py", line 61, in embed_documents worker_1 | embedding_result = self._model_instance.invoke_text_embedding( worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/core/model_manager.py", line 185, in invoke_text_embedding worker_1 | self._round_robin_invoke( worker_1 | File "/app/api/core/model_manager.py", line 334, in _round_robin_invoke worker_1 | return function(*args, **kwargs) worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 46, in invoke worker_1 | raise self._transform_invoke_error(e) worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/core/model_runtime/model_providers/__base/ai_model.py", line 70, in _transform_invoke_error worker_1 | for invoke_error, model_errors in self._invoke_error_mapping.items(): worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/core/model_runtime/model_providers/vertex_ai/_common.py", line 15, in _invoke_error_mapping worker_1 | raise NotImplementedError worker_1 | NotImplementedError worker_1 | 2025-02-17 07:44:15.623 ERROR [ThreadPoolExecutor-17_0] [cached_embedding.py:98] - Failed to embed documents: %s worker_1 | Traceback (most recent call last): worker_1 | File "/app/api/.venv/lib/python3.12/site-packages/google/api_core/grpc_helpers.py", line 76, in error_remapped_callable worker_1 | return callable_(*args, **kwargs) worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/.venv/lib/python3.12/site-packages/grpc/_channel.py", line 1181, in __call__ worker_1 | return _end_unary_response_blocking(state, call, False, None) worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/.venv/lib/python3.12/site-packages/grpc/_channel.py", line 1006, in _end_unary_response_blocking worker_1 | raise _InactiveRpcError(state) # pytype: disable=not-instantiable worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: worker_1 | status = StatusCode.RESOURCE_EXHAUSTED worker_1 | details = "Quota exceeded for aiplatform.googleapis.com/online_prediction_requests_per_base_model with base model: textembedding-gecko. Please submit a quota increase request. https://cloud.google.com/vertex-ai/docs/generative-ai/quotas-genai." worker_1 | debug_error_string = "UNKNOWN:Error received from peer ipv4:74.125.135.95:443 {created_time:"2025-02-17T07:44:15.617492549+00:00", grpc_status:8, grpc_message:"Quota exceeded for aiplatform.googleapis.com/online_prediction_requests_per_base_model with base model: textembedding-gecko. Please submit a quota increase request. https://cloud.google.com/vertex-ai/docs/generative-ai/quotas-genai."}" worker_1 | > worker_1 | worker_1 | The above exception was the direct cause of the following exception: worker_1 | worker_1 | Traceback (most recent call last): worker_1 | File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 44, in invoke worker_1 | return self._invoke(model, credentials, texts, user, input_type) worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/core/model_runtime/model_providers/vertex_ai/text_embedding/text_embedding.py", line 69, in _invoke worker_1 | embeddings_batch, embedding_used_tokens = self._embedding_invoke(client=client, texts=texts) worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/core/model_runtime/model_providers/vertex_ai/text_embedding/text_embedding.py", line 140, in _embedding_invoke worker_1 | response = client.get_embeddings(texts) worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/.venv/lib/python3.12/site-packages/vertexai/language_models/_language_models.py", line 2097, in get_embeddings worker_1 | prediction_response = self._endpoint.predict( worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/.venv/lib/python3.12/site-packages/google/cloud/aiplatform/models.py", line 1597, in predict worker_1 | prediction_response = self._prediction_client.predict( worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/.venv/lib/python3.12/site-packages/google/cloud/aiplatform_v1/services/prediction_service/client.py", line 836, in predict worker_1 | response = rpc( worker_1 | ^^^^ worker_1 | File "/app/api/.venv/lib/python3.12/site-packages/google/api_core/gapic_v1/method.py", line 131, in __call__ worker_1 | return wrapped_func(*args, **kwargs) worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/.venv/lib/python3.12/site-packages/google/api_core/grpc_helpers.py", line 78, in error_remapped_callable worker_1 | raise exceptions.from_grpc_error(exc) from exc worker_1 | google.api_core.exceptions.ResourceExhausted: 429 Quota exceeded for aiplatform.googleapis.com/online_prediction_requests_per_base_model with base model: textembedding-gecko. Please submit a quota increase request. https://cloud.google.com/vertex-ai/docs/generative-ai/quotas-genai. worker_1 | worker_1 | During handling of the above exception, another exception occurred: worker_1 | worker_1 | Traceback (most recent call last): worker_1 | File "/app/api/core/rag/embedding/cached_embedding.py", line 61, in embed_documents worker_1 | embedding_result = self._model_instance.invoke_text_embedding( worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/core/model_manager.py", line 185, in invoke_text_embedding worker_1 | self._round_robin_invoke( worker_1 | File "/app/api/core/model_manager.py", line 334, in _round_robin_invoke worker_1 | return function(*args, **kwargs) worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 46, in invoke worker_1 | raise self._transform_invoke_error(e) worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/core/model_runtime/model_providers/__base/ai_model.py", line 70, in _transform_invoke_error worker_1 | for invoke_error, model_errors in self._invoke_error_mapping.items(): worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/core/model_runtime/model_providers/vertex_ai/_common.py", line 15, in _invoke_error_mapping worker_1 | raise NotImplementedError worker_1 | NotImplementedError worker_1 | 2025-02-17 07:44:15.870 ERROR [ThreadPoolExecutor-17_7] [cached_embedding.py:98] - Failed to embed documents: %s worker_1 | Traceback (most recent call last): worker_1 | File "/app/api/.venv/lib/python3.12/site-packages/google/api_core/grpc_helpers.py", line 76, in error_remapped_callable worker_1 | return callable_(*args, **kwargs) worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/.venv/lib/python3.12/site-packages/grpc/_channel.py", line 1181, in __call__ worker_1 | return _end_unary_response_blocking(state, call, False, None) worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/.venv/lib/python3.12/site-packages/grpc/_channel.py", line 1006, in _end_unary_response_blocking worker_1 | raise _InactiveRpcError(state) # pytype: disable=not-instantiable worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: worker_1 | status = StatusCode.RESOURCE_EXHAUSTED worker_1 | details = "Quota exceeded for aiplatform.googleapis.com/online_prediction_requests_per_base_model with base model: textembedding-gecko. Please submit a quota increase request. https://cloud.google.com/vertex-ai/docs/generative-ai/quotas-genai." worker_1 | debug_error_string = "UNKNOWN:Error received from peer ipv4:74.125.197.95:443 {created_time:"2025-02-17T07:44:15.869419385+00:00", grpc_status:8, grpc_message:"Quota exceeded for aiplatform.googleapis.com/online_prediction_requests_per_base_model with base model: textembedding-gecko. Please submit a quota increase request. https://cloud.google.com/vertex-ai/docs/generative-ai/quotas-genai."}" worker_1 | > worker_1 | worker_1 | The above exception was the direct cause of the following exception: worker_1 | worker_1 | Traceback (most recent call last): worker_1 | File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 44, in invoke worker_1 | return self._invoke(model, credentials, texts, user, input_type) worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/core/model_runtime/model_providers/vertex_ai/text_embedding/text_embedding.py", line 69, in _invoke worker_1 | embeddings_batch, embedding_used_tokens = self._embedding_invoke(client=client, texts=texts) worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/core/model_runtime/model_providers/vertex_ai/text_embedding/text_embedding.py", line 140, in _embedding_invoke worker_1 | response = client.get_embeddings(texts) worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/.venv/lib/python3.12/site-packages/vertexai/language_models/_language_models.py", line 2097, in get_embeddings worker_1 | prediction_response = self._endpoint.predict( worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/.venv/lib/python3.12/site-packages/google/cloud/aiplatform/models.py", line 1597, in predict worker_1 | prediction_response = self._prediction_client.predict( worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/.venv/lib/python3.12/site-packages/google/cloud/aiplatform_v1/services/prediction_service/client.py", line 836, in predict worker_1 | response = rpc( worker_1 | ^^^^ worker_1 | File "/app/api/.venv/lib/python3.12/site-packages/google/api_core/gapic_v1/method.py", line 131, in __call__ worker_1 | return wrapped_func(*args, **kwargs) worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/.venv/lib/python3.12/site-packages/google/api_core/grpc_helpers.py", line 78, in error_remapped_callable worker_1 | raise exceptions.from_grpc_error(exc) from exc worker_1 | google.api_core.exceptions.ResourceExhausted: 429 Quota exceeded for aiplatform.googleapis.com/online_prediction_requests_per_base_model with base model: textembedding-gecko. Please submit a quota increase request. https://cloud.google.com/vertex-ai/docs/generative-ai/quotas-genai. worker_1 | worker_1 | During handling of the above exception, another exception occurred: worker_1 | worker_1 | Traceback (most recent call last): worker_1 | File "/app/api/core/rag/embedding/cached_embedding.py", line 61, in embed_documents worker_1 | embedding_result = self._model_instance.invoke_text_embedding( worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/core/model_manager.py", line 185, in invoke_text_embedding worker_1 | self._round_robin_invoke( worker_1 | File "/app/api/core/model_manager.py", line 334, in _round_robin_invoke worker_1 | return function(*args, **kwargs) worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 46, in invoke worker_1 | raise self._transform_invoke_error(e) worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/core/model_runtime/model_providers/__base/ai_model.py", line 70, in _transform_invoke_error worker_1 | for invoke_error, model_errors in self._invoke_error_mapping.items(): worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/core/model_runtime/model_providers/vertex_ai/_common.py", line 15, in _invoke_error_mapping worker_1 | raise NotImplementedError worker_1 | NotImplementedError worker_1 | 2025-02-17 07:44:15.873 ERROR [Dummy-5] [indexing_runner.py:96] - consume document failed worker_1 | Traceback (most recent call last): worker_1 | File "/app/api/.venv/lib/python3.12/site-packages/google/api_core/grpc_helpers.py", line 76, in error_remapped_callable worker_1 | return callable_(*args, **kwargs) worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/.venv/lib/python3.12/site-packages/grpc/_channel.py", line 1181, in __call__ worker_1 | return _end_unary_response_blocking(state, call, False, None) worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/.venv/lib/python3.12/site-packages/grpc/_channel.py", line 1006, in _end_unary_response_blocking worker_1 | raise _InactiveRpcError(state) # pytype: disable=not-instantiable worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: worker_1 | status = StatusCode.RESOURCE_EXHAUSTED worker_1 | details = "Quota exceeded for aiplatform.googleapis.com/online_prediction_requests_per_base_model with base model: textembedding-gecko. Please submit a quota increase request. https://cloud.google.com/vertex-ai/docs/generative-ai/quotas-genai." worker_1 | debug_error_string = "UNKNOWN:Error received from peer ipv4:74.125.135.95:443 {created_time:"2025-02-17T07:44:15.617492549+00:00", grpc_status:8, grpc_message:"Quota exceeded for aiplatform.googleapis.com/online_prediction_requests_per_base_model with base model: textembedding-gecko. Please submit a quota increase request. https://cloud.google.com/vertex-ai/docs/generative-ai/quotas-genai."}" worker_1 | > worker_1 | worker_1 | The above exception was the direct cause of the following exception: worker_1 | worker_1 | Traceback (most recent call last): worker_1 | File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 44, in invoke worker_1 | return self._invoke(model, credentials, texts, user, input_type) worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/core/model_runtime/model_providers/vertex_ai/text_embedding/text_embedding.py", line 69, in _invoke worker_1 | embeddings_batch, embedding_used_tokens = self._embedding_invoke(client=client, texts=texts) worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/core/model_runtime/model_providers/vertex_ai/text_embedding/text_embedding.py", line 140, in _embedding_invoke worker_1 | response = client.get_embeddings(texts) worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/.venv/lib/python3.12/site-packages/vertexai/language_models/_language_models.py", line 2097, in get_embeddings worker_1 | prediction_response = self._endpoint.predict( worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/.venv/lib/python3.12/site-packages/google/cloud/aiplatform/models.py", line 1597, in predict worker_1 | prediction_response = self._prediction_client.predict( worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/.venv/lib/python3.12/site-packages/google/cloud/aiplatform_v1/services/prediction_service/client.py", line 836, in predict worker_1 | response = rpc( worker_1 | ^^^^ worker_1 | File "/app/api/.venv/lib/python3.12/site-packages/google/api_core/gapic_v1/method.py", line 131, in __call__ worker_1 | return wrapped_func(*args, **kwargs) worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/.venv/lib/python3.12/site-packages/google/api_core/grpc_helpers.py", line 78, in error_remapped_callable worker_1 | raise exceptions.from_grpc_error(exc) from exc worker_1 | google.api_core.exceptions.ResourceExhausted: 429 Quota exceeded for aiplatform.googleapis.com/online_prediction_requests_per_base_model with base model: textembedding-gecko. Please submit a quota increase request. https://cloud.google.com/vertex-ai/docs/generative-ai/quotas-genai. worker_1 | worker_1 | During handling of the above exception, another exception occurred: worker_1 | worker_1 | Traceback (most recent call last): worker_1 | File "/app/api/core/indexing_runner.py", line 80, in run worker_1 | self._load( worker_1 | File "/app/api/core/indexing_runner.py", line 570, in _load worker_1 | tokens += future.result() worker_1 | ^^^^^^^^^^^^^^^ worker_1 | File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 456, in result worker_1 | return self.__get_result() worker_1 | ^^^^^^^^^^^^^^^^^^^ worker_1 | File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result worker_1 | raise self._exception worker_1 | File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 59, in run worker_1 | result = self.fn(*self.args, **self.kwargs) worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/core/indexing_runner.py", line 627, in _process_chunk worker_1 | index_processor.load(dataset, chunk_documents, with_keywords=False) worker_1 | File "/app/api/core/rag/index_processor/processor/paragraph_index_processor.py", line 78, in load worker_1 | vector.create(documents) worker_1 | File "/app/api/core/rag/datasource/vdb/vector_factory.py", line 156, in create worker_1 | embeddings = self._embeddings.embed_documents([document.page_content for document in texts]) worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/core/rag/embedding/cached_embedding.py", line 99, in embed_documents worker_1 | raise ex worker_1 | File "/app/api/core/rag/embedding/cached_embedding.py", line 61, in embed_documents worker_1 | embedding_result = self._model_instance.invoke_text_embedding( worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/core/model_manager.py", line 185, in invoke_text_embedding worker_1 | self._round_robin_invoke( worker_1 | File "/app/api/core/model_manager.py", line 334, in _round_robin_invoke worker_1 | return function(*args, **kwargs) worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 46, in invoke worker_1 | raise self._transform_invoke_error(e) worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/core/model_runtime/model_providers/__base/ai_model.py", line 70, in _transform_invoke_error worker_1 | for invoke_error, model_errors in self._invoke_error_mapping.items(): worker_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^ worker_1 | File "/app/api/core/model_runtime/model_providers/vertex_ai/_common.py", line 15, in _invoke_error_mapping worker_1 | raise NotImplementedError worker_1 | NotImplementedError worker_1 | 2025-02-17 07:44:15.881 INFO [Dummy-5] [retry_document_indexing_task.py:96] - Retry dataset: 0191910d-eced-4e73-b42b-d02bbbc2ed81 latency: 9.137617005966604 worker_1 | 2025-02-17 07:44:15.929 INFO [Dummy-5] [trace.py:128] - Task tasks.retry_document_indexing_task.retry_document_indexing_task[aeb5d7b7-3123-47a4-8026-4a5f04a35026] succeeded in 9.186373426113278s: None ```
yindo added the 👻 feat:rag label 2026-02-21 18:25:06 -05:00
yindo closed this issue 2026-02-21 18:25:06 -05:00
yindo commented 2026-02-21 18:25:06 -05:00
Author
Owner
Copy Link
Copy Source

@lema-founders commented on GitHub (Feb 17, 2025):

@dosu check the docker logs for what's wrong

@lema-founders commented on GitHub (Feb 17, 2025): @dosu check the docker logs for what's wrong
yindo commented 2026-02-21 18:25:06 -05:00
Author
Owner
Copy Link
Copy Source

@crazywoola commented on GitHub (Feb 17, 2025):

@lema-founders

You exceed the rate limit of the provider.

google.api_core.exceptions.ResourceExhausted: 429 Quota exceeded for aiplatform.googleapis.com/online_prediction_requests_per_base_model with base model: textembedding-gecko. Please submit a quota increase request. https://cloud.google.com/vertex-ai/docs/generative-ai/quotas-genai.

Which you should check https://cloud.google.com/vertex-ai/docs/generative-ai/quotas-genai

This problem is not raised by us.

@crazywoola commented on GitHub (Feb 17, 2025): @lema-founders You exceed the rate limit of the provider. ``` google.api_core.exceptions.ResourceExhausted: 429 Quota exceeded for aiplatform.googleapis.com/online_prediction_requests_per_base_model with base model: textembedding-gecko. Please submit a quota increase request. https://cloud.google.com/vertex-ai/docs/generative-ai/quotas-genai. ``` Which you should check https://cloud.google.com/vertex-ai/docs/generative-ai/quotas-genai This problem is not raised by us.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 👻 feat:rag
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#8328
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?