langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-08-28 05:20:07 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

文本分段与清洗失败 #1099

New Issue
Closed
opened 2026-02-21 17:29:59 -05:00 by yindo · 1 comment
yindo commented 2026-02-21 17:29:59 -05:00
Owner
Copy Link
Copy Source

Originally created by @limuan123123 on GitHub (Feb 23, 2024).

Self Checks

  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • Pleas do not modify this template :) and fill in all the required fields.

Provide a description of the help you need

文本分段与清洗上传文档成功，文本分段与清洗失败。
Traceback (most recent call last):
File "/usr/local/dify/api/core/indexing_runner.py", line 67, in run
self._load_segments(dataset, dataset_document, documents)
File "/usr/local/dify/api/core/indexing_runner.py", line 782, in _load_segments
doc_store.add_documents(documents)
File "/usr/local/dify/api/core/docstore/dataset_docstore.py", line 85, in add_documents
raise ValueError("doc must be a Document")
ValueError: doc must be a Document

Originally created by @limuan123123 on GitHub (Feb 23, 2024). ### Self Checks - [X] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [X] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [X] Pleas do not modify this template :) and fill in all the required fields. ### Provide a description of the help you need 文本分段与清洗上传文档成功，文本分段与清洗失败。 Traceback (most recent call last): File "/usr/local/dify/api/core/indexing_runner.py", line 67, in run self._load_segments(dataset, dataset_document, documents) File "/usr/local/dify/api/core/indexing_runner.py", line 782, in _load_segments doc_store.add_documents(documents) File "/usr/local/dify/api/core/docstore/dataset_docstore.py", line 85, in add_documents raise ValueError("doc must be a Document") ValueError: doc must be a Document
yindo closed this issue 2026-02-21 17:29:59 -05:00
yindo commented 2026-02-21 17:29:59 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Feb 23, 2024):

🤖

你好，@limuan123123！很高兴再次见到你。👋😊

根据您提供的错误信息，问题可能出在SegmentService.segment_create_args_validate(args['segments'], document)函数。这个函数期望第二个参数是一个Document对象，但它没有收到一个。在您提供的代码中，document对象是通过DocumentService.get_document(dataset.id, document_id)函数获取的。如果这个函数没有返回一个Document对象（例如，如果提供的ID的文档不存在），那么就会引发NotFound('Document not found.')错误。然而，如果函数返回一个非Document对象，就会出现您看到的错误。

要解决这个问题，您应该确保DocumentService.get_document(dataset.id, document_id)始终返回一个Document对象。如果函数可能返回非Document对象，您应该在函数调用后添加一个类型检查，以确保返回的对象是一个Document。如果不是，您应该引发一个适当的错误。

以下是如何添加类型检查的示例：

document = DocumentService.get_document(dataset.id, document_id)
if not document:
    raise NotFound('Document not found.')
if not isinstance(document, Document):
    raise TypeError('Expected a Document object, but got a different type.')

请将Document替换为您的Document对象的实际类名。

另外，关于Document对象的结构，它是一个继承自pydantic库的BaseModel类的类，用于存储一段文本和相关的元数据。Document对象的结构如下：

  • page_content：这是一个字符串属性，存储文档的实际内容。

  • metadata：这是一个可选的字典属性，存储关于页面内容的任意元数据。这可能包括文档的来源、与其他文档的关系等信息。如果没有提供元数据，它默认为空字典。

这是Document类的代码：

class Document(BaseModel):
    """Class for storing a piece of text and associated metadata."""

    page_content: str

    """Arbitrary metadata about the page content (e.g., source, relationships to other
        documents, etc.).
    """
    metadata: Optional[dict] = Field(default_factory=dict)

希望这些信息能帮助您解决问题。如果还有其他问题，欢迎继续提问。

Sources

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

@dosubot[bot] commented on GitHub (Feb 23, 2024): _🤖_ <!-- Greeting --> 你好，@limuan123123！很高兴再次见到你。👋😊 <!-- Answer --> 根据您提供的错误信息，问题可能出在`SegmentService.segment_create_args_validate(args['segments'], document)`函数。这个函数期望第二个参数是一个Document对象，但它没有收到一个。在您提供的代码中，`document`对象是通过`DocumentService.get_document(dataset.id, document_id)`函数获取的。如果这个函数没有返回一个Document对象（例如，如果提供的ID的文档不存在），那么就会引发`NotFound('Document not found.')`错误。然而，如果函数返回一个非Document对象，就会出现您看到的错误。 要解决这个问题，您应该确保`DocumentService.get_document(dataset.id, document_id)`始终返回一个Document对象。如果函数可能返回非Document对象，您应该在函数调用后添加一个类型检查，以确保返回的对象是一个Document。如果不是，您应该引发一个适当的错误。 以下是如何添加类型检查的示例： ```py document = DocumentService.get_document(dataset.id, document_id) if not document: raise NotFound('Document not found.') if not isinstance(document, Document): raise TypeError('Expected a Document object, but got a different type.') ``` 请将`Document`替换为您的Document对象的实际类名。 另外，关于`Document`对象的结构，它是一个继承自`pydantic`库的`BaseModel`类的类，用于存储一段文本和相关的元数据。`Document`对象的结构如下： - `page_content`：这是一个字符串属性，存储文档的实际内容。 - `metadata`：这是一个可选的字典属性，存储关于页面内容的任意元数据。这可能包括文档的来源、与其他文档的关系等信息。如果没有提供元数据，它默认为空字典。 这是`Document`类的代码： ```py class Document(BaseModel): """Class for storing a piece of text and associated metadata.""" page_content: str """Arbitrary metadata about the page content (e.g., source, relationships to other documents, etc.). """ metadata: Optional[dict] = Field(default_factory=dict) ``` 希望这些信息能帮助您解决问题。如果还有其他问题，欢迎继续提问。 ## Sources - [文档上传使用自定义切割嵌入报错](https://github.com/langgenius/dify/issues/1657) - [添加数据集进行分段时报](https://github.com/langgenius/dify/issues/1225) - [导入文本报错](https://github.com/langgenius/dify/issues/209) - [api/controllers/service_api/dataset/segment.py](https://github.com/langgenius/dify/blob/main/api/controllers/service_api/dataset/segment.py) - [api/core/rag/models/document.py](https://github.com/langgenius/dify/blob/main/api/core/rag/models/document.py) <!-- Dosu Comment Footer --> <details open> <summary> #### About Dosu </summary> This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot. </details>
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#1099
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?