langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-22 20:45:26 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

datasets query error #5515

New Issue
Closed
opened 2026-02-21 18:11:22 -05:00 by yindo · 8 comments
yindo commented 2026-02-21 18:11:22 -05:00
Owner
Copy Link
Copy Source

Originally created by @soulzzz on GitHub (Sep 10, 2024).

Originally assigned to: @crazywoola on GitHub.

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

0.8.0

Cloud or Self Hosted

Self Hosted (Source)

Steps to reproduce

After upgrading the version, some of my knowledge datasets cannot be queried！！！
And i try to create a new dataset and the same error occurred
1.creating a new dataset,and upload 1 file
2.try a hit testing

2024-09-10 09:18:55,802.802 INFO [Thread-218 (process_request_thread)] [_internal.py:97] - 10.8.8.141 - - [10/Sep/2024 09:18:55] "OPTIONS /console/api/datasets/beb7e2e4-fa97-46fc-93c9-26612c3858f0/hit-testing HTTP/1.1" 200 -
2024-09-10 09:18:55,845.845 ERROR [Thread-219 (process_request_thread)] [hit_testing.py:82] - Hit testing failed.
Traceback (most recent call last):
  File "/home/sky/Second/dify/api/controllers/console/datasets/hit_testing.py", line 55, in post
    response = HitTestingService.retrieve(
  File "/home/sky/Second/dify/api/services/hit_testing_service.py", line 38, in retrieve
    all_documents = RetrievalService.retrieve(
  File "/home/sky/Second/dify/api/core/rag/datasource/retrieval_service.py", line 100, in retrieve
    raise Exception(exception_message)
Exception: Error during query: [{'locations': [{'column': 22885, 'line': 1}], 'message': 'Cannot query field "page" on type "Vector_index_beb7e2e4_fa97_46fc_93c9_26612c3858f0_Node".', 'path': None}]
2024-09-10 09:18:55,845.845 ERROR [Thread-219 (process_request_thread)] [app.py:838] - Exception on /console/api/datasets/beb7e2e4-fa97-46fc-93c9-26612c3858f0/hit-testing [POST]
Traceback (most recent call last):
  File "/home/sky/Second/dify/api/controllers/console/datasets/hit_testing.py", line 55, in post
    response = HitTestingService.retrieve(
  File "/home/sky/Second/dify/api/services/hit_testing_service.py", line 38, in retrieve
    all_documents = RetrievalService.retrieve(
  File "/home/sky/Second/dify/api/core/rag/datasource/retrieval_service.py", line 100, in retrieve
    raise Exception(exception_message)
Exception: Error during query: [{'locations': [{'column': 22885, 'line': 1}], 'message': 'Cannot query field "page" on type "Vector_index_beb7e2e4_fa97_46fc_93c9_26612c3858f0_Node".', 'path': None}]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/sky/anaconda3/envs/dify/lib/python3.10/site-packages/flask/app.py", line 880, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/sky/anaconda3/envs/dify/lib/python3.10/site-packages/flask/app.py", line 865, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
  File "/home/sky/anaconda3/envs/dify/lib/python3.10/site-packages/flask_restful/__init__.py", line 489, in wrapper
    resp = resource(*args, **kwargs)
  File "/home/sky/anaconda3/envs/dify/lib/python3.10/site-packages/flask/views.py", line 110, in view
    return current_app.ensure_sync(self.dispatch_request)(**kwargs)  # type: ignore[no-any-return]
  File "/home/sky/anaconda3/envs/dify/lib/python3.10/site-packages/flask_restful/__init__.py", line 604, in dispatch_request
    resp = meth(*args, **kwargs)
  File "/home/sky/Second/dify/api/controllers/console/setup.py", line 65, in decorated
    return view(*args, **kwargs)
  File "/home/sky/Second/dify/api/libs/login.py", line 93, in decorated_view
    return current_app.ensure_sync(func)(*args, **kwargs)
  File "/home/sky/Second/dify/api/controllers/console/wraps.py", line 22, in decorated
    return view(*args, **kwargs)
  File "/home/sky/Second/dify/api/controllers/console/datasets/hit_testing.py", line 83, in post
    raise InternalServerError(str(e))
werkzeug.exceptions.InternalServerError: 500 Internal Server Error: Error during query: [{'locations': [{'column': 22885, 'line': 1}], 'message': 'Cannot query field "page" on type "Vector_index_beb7e2e4_fa97_46fc_93c9_26612c3858f0_Node".', 'path': None}]
2024-09-10 09:18:55,846.846 INFO [Thread-219 (process_request_thread)] [_internal.py:97] - 10.8.8.141 - - [10/Sep/2024 09:18:55] "POST /console/api/datasets/beb7e2e4-fa97-46fc-93c9-26612c3858f0/hit-testing HTTP/1.1" 500 -

✔️ Expected Behavior

queried Successfully

❌ Actual Behavior

queried fail

Originally created by @soulzzz on GitHub (Sep 10, 2024). Originally assigned to: @crazywoola on GitHub. ### Self Checks - [X] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [X] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [X] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [X] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:） - [X] Please do not modify this template :) and fill in all the required fields. ### Dify version 0.8.0 ### Cloud or Self Hosted Self Hosted (Source) ### Steps to reproduce After upgrading the version, some of my knowledge datasets cannot be queried！！！ And i try to create a new dataset and the same error occurred 1.creating a new dataset,and upload 1 file 2.try a hit testing ``` 2024-09-10 09:18:55,802.802 INFO [Thread-218 (process_request_thread)] [_internal.py:97] - 10.8.8.141 - - [10/Sep/2024 09:18:55] "OPTIONS /console/api/datasets/beb7e2e4-fa97-46fc-93c9-26612c3858f0/hit-testing HTTP/1.1" 200 - 2024-09-10 09:18:55,845.845 ERROR [Thread-219 (process_request_thread)] [hit_testing.py:82] - Hit testing failed. Traceback (most recent call last): File "/home/sky/Second/dify/api/controllers/console/datasets/hit_testing.py", line 55, in post response = HitTestingService.retrieve( File "/home/sky/Second/dify/api/services/hit_testing_service.py", line 38, in retrieve all_documents = RetrievalService.retrieve( File "/home/sky/Second/dify/api/core/rag/datasource/retrieval_service.py", line 100, in retrieve raise Exception(exception_message) Exception: Error during query: [{'locations': [{'column': 22885, 'line': 1}], 'message': 'Cannot query field "page" on type "Vector_index_beb7e2e4_fa97_46fc_93c9_26612c3858f0_Node".', 'path': None}] 2024-09-10 09:18:55,845.845 ERROR [Thread-219 (process_request_thread)] [app.py:838] - Exception on /console/api/datasets/beb7e2e4-fa97-46fc-93c9-26612c3858f0/hit-testing [POST] Traceback (most recent call last): File "/home/sky/Second/dify/api/controllers/console/datasets/hit_testing.py", line 55, in post response = HitTestingService.retrieve( File "/home/sky/Second/dify/api/services/hit_testing_service.py", line 38, in retrieve all_documents = RetrievalService.retrieve( File "/home/sky/Second/dify/api/core/rag/datasource/retrieval_service.py", line 100, in retrieve raise Exception(exception_message) Exception: Error during query: [{'locations': [{'column': 22885, 'line': 1}], 'message': 'Cannot query field "page" on type "Vector_index_beb7e2e4_fa97_46fc_93c9_26612c3858f0_Node".', 'path': None}] During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/sky/anaconda3/envs/dify/lib/python3.10/site-packages/flask/app.py", line 880, in full_dispatch_request rv = self.dispatch_request() File "/home/sky/anaconda3/envs/dify/lib/python3.10/site-packages/flask/app.py", line 865, in dispatch_request return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return] File "/home/sky/anaconda3/envs/dify/lib/python3.10/site-packages/flask_restful/__init__.py", line 489, in wrapper resp = resource(*args, **kwargs) File "/home/sky/anaconda3/envs/dify/lib/python3.10/site-packages/flask/views.py", line 110, in view return current_app.ensure_sync(self.dispatch_request)(**kwargs) # type: ignore[no-any-return] File "/home/sky/anaconda3/envs/dify/lib/python3.10/site-packages/flask_restful/__init__.py", line 604, in dispatch_request resp = meth(*args, **kwargs) File "/home/sky/Second/dify/api/controllers/console/setup.py", line 65, in decorated return view(*args, **kwargs) File "/home/sky/Second/dify/api/libs/login.py", line 93, in decorated_view return current_app.ensure_sync(func)(*args, **kwargs) File "/home/sky/Second/dify/api/controllers/console/wraps.py", line 22, in decorated return view(*args, **kwargs) File "/home/sky/Second/dify/api/controllers/console/datasets/hit_testing.py", line 83, in post raise InternalServerError(str(e)) werkzeug.exceptions.InternalServerError: 500 Internal Server Error: Error during query: [{'locations': [{'column': 22885, 'line': 1}], 'message': 'Cannot query field "page" on type "Vector_index_beb7e2e4_fa97_46fc_93c9_26612c3858f0_Node".', 'path': None}] 2024-09-10 09:18:55,846.846 INFO [Thread-219 (process_request_thread)] [_internal.py:97] - 10.8.8.141 - - [10/Sep/2024 09:18:55] "POST /console/api/datasets/beb7e2e4-fa97-46fc-93c9-26612c3858f0/hit-testing HTTP/1.1" 500 - ``` ### ✔️ Expected Behavior queried Successfully ### ❌ Actual Behavior queried fail
yindo added the 🐞 bug label 2026-02-21 18:11:22 -05:00
yindo closed this issue 2026-02-21 18:11:22 -05:00
yindo commented 2026-02-21 18:11:23 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Sep 10, 2024):

It looks like you're encountering a "Cannot query field" error when trying to perform hit testing on your datasets. This issue is often caused by querying a field that is not defined in the dataset fields.

Here are some common causes and solutions for this error:

  1. Field Not Defined: Ensure that the field you are querying, such as page, is defined in the dataset fields. If it is not, you need to add it to the dataset schema.

  2. Typographical Errors: Double-check for any typos in the field name. For example, querying page instead of the correct field name will cause this error.

  3. Incorrect Nesting: If the field is nested, ensure that the query correctly references the nested structure.

Here is an example of correctly defined fields in dataset_query_detail_fields:

dataset_query_detail_fields = {
    "id": fields.String,
    "content": fields.String,
    "source": fields.String,
    "source_app_id": fields.String,
    "created_by_role": fields.String,
    "created_by": fields.String,
    "created_at": TimestampField,
}

Ensure that the field page (or any other field you are querying) is correctly defined and referenced in the dataset fields [1].

Additionally, there are known issues or limitations with the hit-testing endpoint in the current version of Dify, such as:

  1. Dataset Not Found: If the dataset specified by dataset_id does not exist, a NotFound error is raised.
  2. No Permission: If the current user does not have permission to access the dataset, a Forbidden error is raised.
  3. Index Not Initialized: If the index for the dataset is not initialized, a DatasetNotInitializedError is raised.
  4. Provider Token Not Initialized: If the provider token is not initialized, a ProviderNotInitializeError is raised.
  5. Quota Exceeded: If the provider quota is exceeded, a ProviderQuotaExceededError is raised.
  6. Model Not Supported: If the model currently does not support the requested operation, a ProviderModelCurrentlyNotSupportError is raised.
  7. No Embedding or Reranking Model: If no embedding model or reranking model is available, a ProviderNotInitializeError is raised with a specific message.
  8. Invoke Error: If there is an error during the invocation of the model, a CompletionRequestError is raised.
  9. General Errors: Any other exceptions are logged and an InternalServerError is raised [2].

By addressing these potential issues, you should be able to resolve the error and successfully query your datasets.

To continue talking to Dosu, mention @dosu.

@dosubot[bot] commented on GitHub (Sep 10, 2024): <!-- Answer --> It looks like you're encountering a "Cannot query field" error when trying to perform hit testing on your datasets. This issue is often caused by querying a field that is not defined in the dataset fields. Here are some common causes and solutions for this error: 1. **Field Not Defined**: Ensure that the field you are querying, such as `page`, is defined in the dataset fields. If it is not, you need to add it to the dataset schema. 2. **Typographical Errors**: Double-check for any typos in the field name. For example, querying `page` instead of the correct field name will cause this error. 3. **Incorrect Nesting**: If the field is nested, ensure that the query correctly references the nested structure. Here is an example of correctly defined fields in `dataset_query_detail_fields`: ```python dataset_query_detail_fields = { "id": fields.String, "content": fields.String, "source": fields.String, "source_app_id": fields.String, "created_by_role": fields.String, "created_by": fields.String, "created_at": TimestampField, } ``` Ensure that the field `page` (or any other field you are querying) is correctly defined and referenced in the dataset fields <sup>[[1]](https://github.com/langgenius/dify/blob/main/api/fields/dataset_fields.py)</sup>. Additionally, there are known issues or limitations with the `hit-testing` endpoint in the current version of Dify, such as: 1. **Dataset Not Found**: If the dataset specified by `dataset_id` does not exist, a `NotFound` error is raised. 2. **No Permission**: If the current user does not have permission to access the dataset, a `Forbidden` error is raised. 3. **Index Not Initialized**: If the index for the dataset is not initialized, a `DatasetNotInitializedError` is raised. 4. **Provider Token Not Initialized**: If the provider token is not initialized, a `ProviderNotInitializeError` is raised. 5. **Quota Exceeded**: If the provider quota is exceeded, a `ProviderQuotaExceededError` is raised. 6. **Model Not Supported**: If the model currently does not support the requested operation, a `ProviderModelCurrentlyNotSupportError` is raised. 7. **No Embedding or Reranking Model**: If no embedding model or reranking model is available, a `ProviderNotInitializeError` is raised with a specific message. 8. **Invoke Error**: If there is an error during the invocation of the model, a `CompletionRequestError` is raised. 9. **General Errors**: Any other exceptions are logged and an `InternalServerError` is raised <sup>[[2]](https://github.com/langgenius/dify/blob/main/api/controllers/console/datasets/hit_testing.py)</sup>. By addressing these potential issues, you should be able to resolve the error and successfully query your datasets. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
yindo commented 2026-02-21 18:11:23 -05:00
Author
Owner
Copy Link
Copy Source

@AAEE86 commented on GitHub (Sep 10, 2024):

Error during query: [{'locations': [{'column': 161, 'line': 1}], 'message': 'Cannot query field "page" on type "Vector_index_ed9112e3_6883_4e92_bb39_13f56f4dbe18_Node".', 'path': None}]; Error during query: [{'locations': [{'column': 17089, 'line': 1}], 'message': 'Cannot query field "page" on type "Vector_index_ed9112e3_6883_4e92_bb39_13f56f4dbe18_Node".', 'path': None}]

@AAEE86 commented on GitHub (Sep 10, 2024): `Error during query: [{'locations': [{'column': 161, 'line': 1}], 'message': 'Cannot query field "page" on type "Vector_index_ed9112e3_6883_4e92_bb39_13f56f4dbe18_Node".', 'path': None}]; Error during query: [{'locations': [{'column': 17089, 'line': 1}], 'message': 'Cannot query field "page" on type "Vector_index_ed9112e3_6883_4e92_bb39_13f56f4dbe18_Node".', 'path': None}]`
yindo commented 2026-02-21 18:11:23 -05:00
Author
Owner
Copy Link
Copy Source

@Howe829 commented on GitHub (Sep 10, 2024):

I'll put some related issues here #8117 #8175

@Howe829 commented on GitHub (Sep 10, 2024): I'll put some related issues here #8117 #8175
yindo commented 2026-02-21 18:11:23 -05:00
Author
Owner
Copy Link
Copy Source

@AAEE86 commented on GitHub (Sep 10, 2024):

Error during query: [{'locations': [{'column': 22888, 'line': 1}], 'message': 'Cannot query field "page" on type "Vector_index_ed9112e3_6883_4e92_bb39_13f56f4dbe18_Node".', 'path': None}]



During handling of the above exception, another exception occurred:



Traceback (most recent call last):

  File "/app/api/.venv/lib/python3.10/site-packages/flask/app.py", line 880, in full_dispatch_request

    rv = self.dispatch_request()

  File "/app/api/.venv/lib/python3.10/site-packages/flask/app.py", line 865, in dispatch_request

    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]

  File "/app/api/.venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 489, in wrapper

    resp = resource(*args, **kwargs)

  File "/app/api/.venv/lib/python3.10/site-packages/flask/views.py", line 110, in view

    return current_app.ensure_sync(self.dispatch_request)(**kwargs)  # type: ignore[no-any-return]

  File "/app/api/.venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 604, in dispatch_request

    resp = meth(*args, **kwargs)

  File "/app/api/controllers/console/setup.py", line 65, in decorated

    return view(*args, **kwargs)

  File "/app/api/libs/login.py", line 93, in decorated_view

    return current_app.ensure_sync(func)(*args, **kwargs)

  File "/app/api/controllers/console/wraps.py", line 22, in decorated

    return view(*args, **kwargs)

  File "/app/api/controllers/console/datasets/hit_testing.py", line 83, in post

    raise InternalServerError(str(e))

werkzeug.exceptions.InternalServerError: 500 Internal Server Error: Error during query: [{'locations': [{'column': 150, 'line': 1}], 'message': 'Cannot query field "page" on type "Vector_index_ed9112e3_6883_4e92_bb39_13f56f4dbe18_Node".', 'path': None}];

Error during query: [{'locations': [{'column': 22888, 'line': 1}], 'message': 'Cannot query field "page" on type "Vector_index_ed9112e3_6883_4e92_bb39_13f56f4dbe18_Node".', 'path': None}]
@AAEE86 commented on GitHub (Sep 10, 2024): ``` Error during query: [{'locations': [{'column': 22888, 'line': 1}], 'message': 'Cannot query field "page" on type "Vector_index_ed9112e3_6883_4e92_bb39_13f56f4dbe18_Node".', 'path': None}] During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/app/api/.venv/lib/python3.10/site-packages/flask/app.py", line 880, in full_dispatch_request rv = self.dispatch_request() File "/app/api/.venv/lib/python3.10/site-packages/flask/app.py", line 865, in dispatch_request return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return] File "/app/api/.venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 489, in wrapper resp = resource(*args, **kwargs) File "/app/api/.venv/lib/python3.10/site-packages/flask/views.py", line 110, in view return current_app.ensure_sync(self.dispatch_request)(**kwargs) # type: ignore[no-any-return] File "/app/api/.venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 604, in dispatch_request resp = meth(*args, **kwargs) File "/app/api/controllers/console/setup.py", line 65, in decorated return view(*args, **kwargs) File "/app/api/libs/login.py", line 93, in decorated_view return current_app.ensure_sync(func)(*args, **kwargs) File "/app/api/controllers/console/wraps.py", line 22, in decorated return view(*args, **kwargs) File "/app/api/controllers/console/datasets/hit_testing.py", line 83, in post raise InternalServerError(str(e)) werkzeug.exceptions.InternalServerError: 500 Internal Server Error: Error during query: [{'locations': [{'column': 150, 'line': 1}], 'message': 'Cannot query field "page" on type "Vector_index_ed9112e3_6883_4e92_bb39_13f56f4dbe18_Node".', 'path': None}]; Error during query: [{'locations': [{'column': 22888, 'line': 1}], 'message': 'Cannot query field "page" on type "Vector_index_ed9112e3_6883_4e92_bb39_13f56f4dbe18_Node".', 'path': None}] ```
yindo commented 2026-02-21 18:11:23 -05:00
Author
Owner
Copy Link
Copy Source

@crazywoola commented on GitHub (Sep 10, 2024):

I think this is introduced in this pr https://github.com/langgenius/dify/pull/7749

@crazywoola commented on GitHub (Sep 10, 2024): I think this is introduced in this pr https://github.com/langgenius/dify/pull/7749
yindo commented 2026-02-21 18:11:23 -05:00
Author
Owner
Copy Link
Copy Source

@Howe829 commented on GitHub (Sep 10, 2024):

I think this is introduced in this pr #7749

No wonder, I tried the pdf file the hit-testing worked well, but the txt file failed.

@Howe829 commented on GitHub (Sep 10, 2024): > I think this is introduced in this pr #7749 No wonder, I tried the pdf file the hit-testing worked well, but the txt file failed.
yindo commented 2026-02-21 18:11:23 -05:00
Author
Owner
Copy Link
Copy Source

@soulzzz commented on GitHub (Sep 10, 2024):

I think this is introduced in this pr #7749

I think this is the root cause of the problem，i'm using weaviate DB as default DB

@soulzzz commented on GitHub (Sep 10, 2024): > I think this is introduced in this pr #7749 I think this is the root cause of the problem，i'm using weaviate DB as default DB
yindo commented 2026-02-21 18:11:24 -05:00
Author
Owner
Copy Link
Copy Source

@Howe829 commented on GitHub (Sep 10, 2024):

https://github.com/langgenius/dify/blob/86f7f245e4331d6025496a27a80fd4bcc72e0802/api/core/rag/datasource/vdb/vector_factory.py#L33
remove the 'page' from attributes it will be solved.

@Howe829 commented on GitHub (Sep 10, 2024): https://github.com/langgenius/dify/blob/86f7f245e4331d6025496a27a80fd4bcc72e0802/api/core/rag/datasource/vdb/vector_factory.py#L33 remove the 'page' from attributes it will be solved.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#5515
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?