langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-22 04:25:29 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

preview chunk error #12137

New Issue
Closed
opened 2026-02-21 19:05:58 -05:00 by yindo · 6 comments
yindo commented 2026-02-21 19:05:58 -05:00
Owner
Copy Link
Copy Source

Originally created by @murphy-zhul on GitHub (Mar 25, 2025).

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:)
  • Please do not modify this template :) and fill in all the required fields.

Dify version

1.0.3

Cloud or Self Hosted

Self Hosted (Source)

Steps to reproduce

In the second step of creating the knowledge base, the economic index mode is selected to preview the block interface error
{
"code": "indexing_estimate_error",
"message": "Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/data/dify1.0.1/api/core/model_runtime/model_providers/__base/tokenizers/gpt2'. Use repo_type argument if needed.",
"status": 500
}
and the api error is as follows:
controllers.console.datasets.error.IndexingEstimateError: 500 Internal Server Error: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/data/dify1.0.1/api/core/model_runtime/model_providers/__base/tokenizers/gpt2'. Use repo_type argument if needed.

✔️ Expected Behavior

No response

❌ Actual Behavior

No response

Originally created by @murphy-zhul on GitHub (Mar 25, 2025). ### Self Checks - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:) - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version 1.0.3 ### Cloud or Self Hosted Self Hosted (Source) ### Steps to reproduce In the second step of creating the knowledge base, the economic index mode is selected to preview the block interface error { "code": "indexing_estimate_error", "message": "Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/data/dify1.0.1/api/core/model_runtime/model_providers/__base/tokenizers/gpt2'. Use `repo_type` argument if needed.", "status": 500 } and the api error is as follows: controllers.console.datasets.error.IndexingEstimateError: 500 Internal Server Error: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/data/dify1.0.1/api/core/model_runtime/model_providers/__base/tokenizers/gpt2'. Use `repo_type` argument if needed. ### ✔️ Expected Behavior _No response_ ### ❌ Actual Behavior _No response_
yindo added the 🐞 bug🌚 invalid labels 2026-02-21 19:05:58 -05:00
yindo closed this issue 2026-02-21 19:05:58 -05:00
yindo commented 2026-02-21 19:05:59 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Mar 25, 2025):

@muphy-zhul Hello! I'm here to help you with bug reports, answer questions, and assist you while waiting for a human maintainer. Let me know how I can assist you!

The error message you're encountering, "Repo id must be in the form 'repo_name' or 'namespace/repo_name'," suggests that there is an issue with the format of the repository identifier being used. This error typically occurs when the repository ID does not match the expected format, which should be either 'repo_name' or 'namespace/repo_name'.

To resolve this issue, ensure that the repository ID you are using adheres to the expected format. If your use case requires a different format, consider using the repo_type argument as suggested in the error message. This might involve checking the configuration or the code where the repository ID is being set to ensure it matches the required format. Unfortunately, I couldn't find specific details on where this logic is implemented in the Dify codebase, so you might need to review the relevant sections of your setup or code to identify where the incorrect format is being applied.

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other

@dosubot[bot] commented on GitHub (Mar 25, 2025): <!-- Greeting --> @muphy-zhul Hello! I'm here to help you with bug reports, answer questions, and assist you while waiting for a human maintainer. Let me know how I can assist you! <!-- Answer --> The error message you're encountering, "Repo id must be in the form 'repo_name' or 'namespace/repo_name'," suggests that there is an issue with the format of the repository identifier being used. This error typically occurs when the repository ID does not match the expected format, which should be either 'repo_name' or 'namespace/repo_name'. To resolve this issue, ensure that the repository ID you are using adheres to the expected format. If your use case requires a different format, consider using the `repo_type` argument as suggested in the error message. This might involve checking the configuration or the code where the repository ID is being set to ensure it matches the required format. Unfortunately, I couldn't find specific details on where this logic is implemented in the Dify codebase, so you might need to review the relevant sections of your setup or code to identify where the incorrect format is being applied. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/4d4a9d6b-de7b-400d-851f-695325b9b1a9?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/4d4a9d6b-de7b-400d-851f-695325b9b1a9?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/4d4a9d6b-de7b-400d-851f-695325b9b1a9?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/4d4a9d6b-de7b-400d-851f-695325b9b1a9?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/4d4a9d6b-de7b-400d-851f-695325b9b1a9?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/4d4a9d6b-de7b-400d-851f-695325b9b1a9?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/4d4a9d6b-de7b-400d-851f-695325b9b1a9?feedback_type=other)</sup>
yindo commented 2026-02-21 19:05:59 -05:00
Author
Owner
Copy Link
Copy Source

@crazywoola commented on GitHub (Mar 25, 2025):

Can you provide more information?

https://github.com/langgenius/dify/blob/0277a37fcad5ad86aeb239485c27fffd5cd90043/CONTRIBUTING.md#L23-L33

Also, it seems you are using GitHub plugin?

@crazywoola commented on GitHub (Mar 25, 2025): Can you provide more information? https://github.com/langgenius/dify/blob/0277a37fcad5ad86aeb239485c27fffd5cd90043/CONTRIBUTING.md#L23-L33 Also, it seems you are using GitHub plugin?
yindo commented 2026-02-21 19:05:59 -05:00
Author
Owner
Copy Link
Copy Source

@murphy-zhul commented on GitHub (Mar 25, 2025):

Can you provide more information?

dify/CONTRIBUTING.md

Lines 23 to 33 in 0277a37

Bug reports

Important

Please make sure to include the following information when submitting a bug report:

  • A clear and descriptive title
  • A detailed description of the bug, including any error messages
  • Steps to reproduce the bug
  • Expected behavior
  • Logs, if available, for backend issues, this is really important, you can find them in docker-compose logs
  • Screenshots or videos, if applicable
    Also, it seems you are using GitHub plugin?

this is my first time deploy dify and i did not set the model,just create konwledge with docx file and i think file is uploaded succussfully . in the step 2,i use the economy index model to chunk

the infomation is as follow:

2025-03-26 02:37:35,204.204 DEBUG [Thread-2559 (process_request_thread)] [opendal_storage.py:79] - file upload_files/3bde6fef-003e-4d02-88be-ad2a6e172dbc/8d2f5c89-8f1f-43ee-9afb-6858c0db1da3.docx checked
2025-03-26 02:37:35,205.205 DEBUG [Thread-2559 (process_request_thread)] [opendal_storage.py:71] - file upload_files/3bde6fef-003e-4d02-88be-ad2a6e172dbc/8d2f5c89-8f1f-43ee-9afb-6858c0db1da3.docx downloaded to /tmp/tmp2lybydrn/x7o8qvyb.docx
2025-03-26 02:37:35,222.222 DEBUG [Thread-2559 (process_request_thread)] [connectionpool.py:1049] - Starting new HTTPS connection (1): openaipublic.blob.core.windows.net:443
2025-03-26 02:37:35,240.240 ERROR [Thread-2559 (process_request_thread)] [app.py:875] - Exception on /console/api/datasets/indexing-estimate [POST]
Traceback (most recent call last):
  File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/urllib3/connection.py", line 198, in _new_conn
    sock = connection.create_connection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/urllib3/util/connection.py", line 85, in create_connection
    raise err
  File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/urllib3/connectionpool.py", line 488, in _make_request
    raise new_e
  File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/urllib3/connectionpool.py", line 464, in _make_request
    self._validate_conn(conn)
  File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/urllib3/connectionpool.py", line 1093, in _validate_conn
    conn.connect()
  File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/urllib3/connection.py", line 704, in connect
    self.sock = sock = self._new_conn()
                       ^^^^^^^^^^^^^^^^
  File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/urllib3/connection.py", line 213, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7fce76b48cb0>: Failed to establish a new connection: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/urllib3/connectionpool.py", line 841, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='openaipublic.blob.core.windows.net', port=443): Max retries exceeded with url: /gpt-2/encodings/main/vocab.bpe (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fce76b48cb0>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/dify1.0.1/api/core/model_runtime/model_providers/__base/tokenizers/gpt2_tokenzier.py", line 40, in get_encoder
    _tokenizer = tiktoken.get_encoding("gpt2")
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/tiktoken/registry.py", line 86, in get_encoding
    enc = Encoding(**constructor())
                     ^^^^^^^^^^^^^
  File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/tiktoken_ext/openai_public.py", line 18, in gpt2
    mergeable_ranks = data_gym_to_mergeable_bpe_ranks(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/tiktoken/load.py", line 103, in data_gym_to_mergeable_bpe_ranks
    vocab_bpe_contents = read_file_cached(vocab_bpe_file, vocab_bpe_hash).decode()
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/tiktoken/load.py", line 63, in read_file_cached
    contents = read_file(blobpath)
               ^^^^^^^^^^^^^^^^^^^
  File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/tiktoken/load.py", line 24, in read_file
    resp = requests.get(blobpath)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/requests/adapters.py", line 519, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='openaipublic.blob.core.windows.net', port=443): Max retries exceeded with url: /gpt-2/encodings/main/vocab.bpe (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fce76b48cb0>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/dify1.0.1/api/controllers/console/datasets/datasets.py", line 464, in post
    response = indexing_runner.indexing_estimate(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/dify1.0.1/api/core/indexing_runner.py", line 290, in indexing_estimate
    documents = index_processor.transform(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/dify1.0.1/api/core/rag/index_processor/processor/paragraph_index_processor.py", line 58, in transform
    document_nodes = splitter.split_documents([document])
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/dify1.0.1/api/core/rag/splitter/text_splitter.py", line 96, in split_documents
    return self.create_documents(texts, metadatas=metadatas)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/dify1.0.1/api/core/rag/splitter/text_splitter.py", line 81, in create_documents
    for chunk in self.split_text(text):
                 ^^^^^^^^^^^^^^^^^^^^^
  File "/data/dify1.0.1/api/core/rag/splitter/fixed_text_splitter.py", line 68, in split_text
    chunks_lengths = self._length_function(chunks)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/dify1.0.1/api/core/rag/splitter/fixed_text_splitter.py", line 40, in _token_encoder
    return [GPT2Tokenizer.get_num_tokens(text) for text in texts]
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/dify1.0.1/api/core/model_runtime/model_providers/__base/tokenizers/gpt2_tokenzier.py", line 28, in get_num_tokens
    return GPT2Tokenizer._get_num_tokens_by_gpt2(text)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/dify1.0.1/api/core/model_runtime/model_providers/__base/tokenizers/gpt2_tokenzier.py", line 17, in _get_num_tokens_by_gpt2
    _tokenizer = GPT2Tokenizer.get_encoder()
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/dify1.0.1/api/core/model_runtime/model_providers/__base/tokenizers/gpt2_tokenzier.py", line 48, in get_encoder
    _tokenizer = TransformerGPT2Tokenizer.from_pretrained(gpt2_tokenizer_path)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 1947, in from_pretrained
    resolved_config_file = cached_file(
                           ^^^^^^^^^^^^
  File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/transformers/utils/hub.py", line 430, in cached_file
    resolved_file = hf_hub_download(
                    ^^^^^^^^^^^^^^^^
  File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py", line 106, in _inner_fn
    validate_repo_id(arg_value)
  File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py", line 154, in validate_repo_id
    raise HFValidationError(
huggingface_hub.errors.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/data/dify1.0.1/api/core/model_runtime/model_providers/__base/tokenizers/gpt2'. Use `repo_type` argument if needed.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/flask/app.py", line 917, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/flask/app.py", line 902, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/flask_restful/__init__.py", line 489, in wrapper
    resp = resource(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/flask/views.py", line 110, in view
    return current_app.ensure_sync(self.dispatch_request)(**kwargs)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/flask_restful/__init__.py", line 604, in dispatch_request
    resp = meth(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/data/dify1.0.1/api/controllers/console/wraps.py", line 187, in decorated
    return view(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/data/dify1.0.1/api/libs/login.py", line 94, in decorated_view
    return current_app.ensure_sync(func)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/dify1.0.1/api/controllers/console/wraps.py", line 30, in decorated
    return view(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/data/dify1.0.1/api/controllers/console/datasets/datasets.py", line 480, in post
    raise IndexingEstimateError(str(e))
controllers.console.datasets.error.IndexingEstimateError: 500 Internal Server Error: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/data/dify1.0.1/api/core/model_runtime/model_providers/__base/tokenizers/gpt2'. Use `repo_type` argument if needed.
2025-03-26 02:37:35,246.246 INFO [Thread-2559 (process_request_thread)] [_internal.py:97] - 10.202.44.173 - - [26/Mar/2025 02:37:35] "POST /console/api/datasets/indexing-estimate HTTP/1.1" 500
@murphy-zhul commented on GitHub (Mar 25, 2025): > Can you provide more information? > > [dify/CONTRIBUTING.md](https://github.com/langgenius/dify/blob/0277a37fcad5ad86aeb239485c27fffd5cd90043/CONTRIBUTING.md#L23-L33) > > Lines 23 to 33 in [0277a37](/langgenius/dify/commit/0277a37fcad5ad86aeb239485c27fffd5cd90043) > > ### Bug reports > > > [!IMPORTANT] > > Please make sure to include the following information when submitting a bug report: > > - A clear and descriptive title > - A detailed description of the bug, including any error messages > - Steps to reproduce the bug > - Expected behavior > - **Logs**, if available, for backend issues, this is really important, you can find them in docker-compose logs > - Screenshots or videos, if applicable > Also, it seems you are using GitHub plugin? this is my first time deploy dify and i did not set the model,just create konwledge with docx file and i think file is uploaded succussfully . in the step 2,i use the economy index model to chunk the infomation is as follow: ``` 2025-03-26 02:37:35,204.204 DEBUG [Thread-2559 (process_request_thread)] [opendal_storage.py:79] - file upload_files/3bde6fef-003e-4d02-88be-ad2a6e172dbc/8d2f5c89-8f1f-43ee-9afb-6858c0db1da3.docx checked 2025-03-26 02:37:35,205.205 DEBUG [Thread-2559 (process_request_thread)] [opendal_storage.py:71] - file upload_files/3bde6fef-003e-4d02-88be-ad2a6e172dbc/8d2f5c89-8f1f-43ee-9afb-6858c0db1da3.docx downloaded to /tmp/tmp2lybydrn/x7o8qvyb.docx 2025-03-26 02:37:35,222.222 DEBUG [Thread-2559 (process_request_thread)] [connectionpool.py:1049] - Starting new HTTPS connection (1): openaipublic.blob.core.windows.net:443 2025-03-26 02:37:35,240.240 ERROR [Thread-2559 (process_request_thread)] [app.py:875] - Exception on /console/api/datasets/indexing-estimate [POST] Traceback (most recent call last): File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/urllib3/connection.py", line 198, in _new_conn sock = connection.create_connection( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/urllib3/util/connection.py", line 85, in create_connection raise err File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/urllib3/util/connection.py", line 73, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 111] Connection refused The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/urllib3/connectionpool.py", line 787, in urlopen response = self._make_request( ^^^^^^^^^^^^^^^^^^^ File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/urllib3/connectionpool.py", line 488, in _make_request raise new_e File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/urllib3/connectionpool.py", line 464, in _make_request self._validate_conn(conn) File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/urllib3/connectionpool.py", line 1093, in _validate_conn conn.connect() File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/urllib3/connection.py", line 704, in connect self.sock = sock = self._new_conn() ^^^^^^^^^^^^^^^^ File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/urllib3/connection.py", line 213, in _new_conn raise NewConnectionError( urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7fce76b48cb0>: Failed to establish a new connection: [Errno 111] Connection refused The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/requests/adapters.py", line 486, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/urllib3/connectionpool.py", line 841, in urlopen retries = retries.increment( ^^^^^^^^^^^^^^^^^^ File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='openaipublic.blob.core.windows.net', port=443): Max retries exceeded with url: /gpt-2/encodings/main/vocab.bpe (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fce76b48cb0>: Failed to establish a new connection: [Errno 111] Connection refused')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/data/dify1.0.1/api/core/model_runtime/model_providers/__base/tokenizers/gpt2_tokenzier.py", line 40, in get_encoder _tokenizer = tiktoken.get_encoding("gpt2") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/tiktoken/registry.py", line 86, in get_encoding enc = Encoding(**constructor()) ^^^^^^^^^^^^^ File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/tiktoken_ext/openai_public.py", line 18, in gpt2 mergeable_ranks = data_gym_to_mergeable_bpe_ranks( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/tiktoken/load.py", line 103, in data_gym_to_mergeable_bpe_ranks vocab_bpe_contents = read_file_cached(vocab_bpe_file, vocab_bpe_hash).decode() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/tiktoken/load.py", line 63, in read_file_cached contents = read_file(blobpath) ^^^^^^^^^^^^^^^^^^^ File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/tiktoken/load.py", line 24, in read_file resp = requests.get(blobpath) ^^^^^^^^^^^^^^^^^^^^^^ File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/requests/api.py", line 73, in get return request("get", url, params=params, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/requests/api.py", line 59, in request return session.request(method=method, url=url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/requests/adapters.py", line 519, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='openaipublic.blob.core.windows.net', port=443): Max retries exceeded with url: /gpt-2/encodings/main/vocab.bpe (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fce76b48cb0>: Failed to establish a new connection: [Errno 111] Connection refused')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/data/dify1.0.1/api/controllers/console/datasets/datasets.py", line 464, in post response = indexing_runner.indexing_estimate( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/dify1.0.1/api/core/indexing_runner.py", line 290, in indexing_estimate documents = index_processor.transform( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/dify1.0.1/api/core/rag/index_processor/processor/paragraph_index_processor.py", line 58, in transform document_nodes = splitter.split_documents([document]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/dify1.0.1/api/core/rag/splitter/text_splitter.py", line 96, in split_documents return self.create_documents(texts, metadatas=metadatas) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/dify1.0.1/api/core/rag/splitter/text_splitter.py", line 81, in create_documents for chunk in self.split_text(text): ^^^^^^^^^^^^^^^^^^^^^ File "/data/dify1.0.1/api/core/rag/splitter/fixed_text_splitter.py", line 68, in split_text chunks_lengths = self._length_function(chunks) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/dify1.0.1/api/core/rag/splitter/fixed_text_splitter.py", line 40, in _token_encoder return [GPT2Tokenizer.get_num_tokens(text) for text in texts] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/dify1.0.1/api/core/model_runtime/model_providers/__base/tokenizers/gpt2_tokenzier.py", line 28, in get_num_tokens return GPT2Tokenizer._get_num_tokens_by_gpt2(text) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/dify1.0.1/api/core/model_runtime/model_providers/__base/tokenizers/gpt2_tokenzier.py", line 17, in _get_num_tokens_by_gpt2 _tokenizer = GPT2Tokenizer.get_encoder() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/dify1.0.1/api/core/model_runtime/model_providers/__base/tokenizers/gpt2_tokenzier.py", line 48, in get_encoder _tokenizer = TransformerGPT2Tokenizer.from_pretrained(gpt2_tokenizer_path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 1947, in from_pretrained resolved_config_file = cached_file( ^^^^^^^^^^^^ File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/transformers/utils/hub.py", line 430, in cached_file resolved_file = hf_hub_download( ^^^^^^^^^^^^^^^^ File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py", line 106, in _inner_fn validate_repo_id(arg_value) File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py", line 154, in validate_repo_id raise HFValidationError( huggingface_hub.errors.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/data/dify1.0.1/api/core/model_runtime/model_providers/__base/tokenizers/gpt2'. Use `repo_type` argument if needed. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/flask/app.py", line 917, in full_dispatch_request rv = self.dispatch_request() ^^^^^^^^^^^^^^^^^^^^^^^ File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/flask/app.py", line 902, in dispatch_request return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/flask_restful/__init__.py", line 489, in wrapper resp = resource(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/flask/views.py", line 110, in view return current_app.ensure_sync(self.dispatch_request)(**kwargs) # type: ignore[no-any-return] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/anaconda3/envs/dify3.12/lib/python3.12/site-packages/flask_restful/__init__.py", line 604, in dispatch_request resp = meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/data/dify1.0.1/api/controllers/console/wraps.py", line 187, in decorated return view(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/data/dify1.0.1/api/libs/login.py", line 94, in decorated_view return current_app.ensure_sync(func)(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/dify1.0.1/api/controllers/console/wraps.py", line 30, in decorated return view(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/data/dify1.0.1/api/controllers/console/datasets/datasets.py", line 480, in post raise IndexingEstimateError(str(e)) controllers.console.datasets.error.IndexingEstimateError: 500 Internal Server Error: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/data/dify1.0.1/api/core/model_runtime/model_providers/__base/tokenizers/gpt2'. Use `repo_type` argument if needed. 2025-03-26 02:37:35,246.246 INFO [Thread-2559 (process_request_thread)] [_internal.py:97] - 10.202.44.173 - - [26/Mar/2025 02:37:35] "POST /console/api/datasets/indexing-estimate HTTP/1.1" 500 ```
yindo commented 2026-02-21 19:05:59 -05:00
Author
Owner
Copy Link
Copy Source

@crazywoola commented on GitHub (Mar 25, 2025):

You can not access this site openaipublic.blob.core.windows.net.

@crazywoola commented on GitHub (Mar 25, 2025): You can not access this site `openaipublic.blob.core.windows.net`.
yindo commented 2026-02-21 19:05:59 -05:00
Author
Owner
Copy Link
Copy Source

@crazywoola commented on GitHub (Mar 25, 2025):

See https://github.com/langgenius/dify/issues/14887 this thread instead.

@crazywoola commented on GitHub (Mar 25, 2025): See https://github.com/langgenius/dify/issues/14887 this thread instead.
yindo commented 2026-02-21 19:05:59 -05:00
Author
Owner
Copy Link
Copy Source

@murphy-zhul commented on GitHub (Mar 31, 2025):

if someone encountered this problem when deploying with source code,u can try it as follows:
manual download gpt2 tokenizer files on https://huggingface.co/models,and put this folder under /api/core/model_runtime/model_providers/__base/tokenizers.

@murphy-zhul commented on GitHub (Mar 31, 2025): if someone encountered this problem when deploying with source code,u can try it as follows: manual download gpt2 tokenizer files on https://huggingface.co/models,and put this folder under /api/core/model_runtime/model_providers/__base/tokenizers.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🌚 invalid 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#12137
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?