[DOCS]: How to invoke the knowledge base pipeline through API when creating a document via file? #260

New Issue

2026-02-16T05:19:38-05:00

yindo commented

2026-02-16 05:19:38 -05:00

Originally created by @xiaohe123456 on GitHub (Oct 31, 2025).

Type of Documentation Change

Other

Documentation Page URL or Path

No response

Current Content

No response

Suggested Changes

In the dify-2.0.0-beta.1 version, after the knowledge base pipeline is created and released, how can documents be created through the API?

Reason for Change

No response

Code of Conduct

I agree to follow Dify's documentation contribution guidelines
I have searched for existing issues search for existing issues, including closed ones.
I confirm that I am using English to submit this report, otherwise it will be closed.
【中文用户 & Non English User】请使用英语提交，否则会被关闭：）
Please do not modify this template :) and fill in all the required fields.

Originally created by @xiaohe123456 on GitHub (Oct 31, 2025). ### Type of Documentation Change Other ### Documentation Page URL or Path _No response_ ### Current Content _No response_ ### Suggested Changes In the dify-2.0.0-beta.1 version, after the knowledge base pipeline is created and released, how can documents be created through the API? ### Reason for Change _No response_ ### Code of Conduct - [x] I agree to follow Dify's documentation [contribution guidelines](https://github.com/langgenius/dify/blob/0277a37fcad5ad86aeb239485c27fffd5cd90043/CONTRIBUTING.md) - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report, otherwise it will be closed. - [x] 【中文用户 & Non English User】请使用英语提交，否则会被关闭：） - [x] Please do not modify this template :) and fill in all the required fields.

yindo added the documentation question labels 2026-02-16 05:19:38 -05:00

yindo closed this issue

2026-02-16 05:19:38 -05:00

yindo commented

2026-02-16 05:19:39 -05:00

@dosubot[bot] commented on GitHub (Oct 31, 2025):

Hi @xiaohe123456! I'm Dosu and I’m helping the dify-docs team.

To create a document via file upload and invoke the knowledge base pipeline through the API in Dify v2.0.0-beta.1, use the endpoint:

POST /v1/datasets/{dataset_id}/document/create-by-file

Send a multipart/form-data request with two fields:

file: the file to upload
data: a JSON string with metadata, such as indexing_technique, process_rule, and optionally doc_form for hierarchical mode.

Include the header:

Authorization: Bearer {api_key}

Example cURL:

curl --location --request POST 'https://api.dify.ai/v1/datasets/{dataset_id}/document/create-by-file' \
--header 'Authorization: Bearer {api_key}' \
--form 'data="{\"indexing_technique\":\"high_quality\",\"process_rule\":{\"mode\":\"hierarchical\"},\"doc_form\":\"hierarchical_model\"}";type=text/plain' \
--form 'file=@"path/to/your/file.md"'

After upload, Dify will automatically process and parse the file in the knowledge base pipeline. For advanced pipeline features (like custom extractors), API support may be limited compared to the Web UI, but using a user-scoped API token and matching your process_rule to the pipeline config maximizes compatibility. For more details and examples, see this issue and this discussion.

If you use the Python SDK, the KnowledgeBaseClient.upload_pipeline_file() method is designed for this purpose reference.

For large files (>800KB), use a Python code node instead of the HTTP request node in Dify workflows reference.

To reply, just mention @dosu.

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

@dosubot[bot] commented on GitHub (Oct 31, 2025):  Hi @xiaohe123456! I'm [Dosu](https://go.dosu.dev/dosubot) and I’m helping the dify-docs team.  To create a document via file upload and invoke the knowledge base pipeline through the API in Dify v2.0.0-beta.1, use the endpoint: ``` POST /v1/datasets/{dataset_id}/document/create-by-file ``` Send a multipart/form-data request with two fields: - `file`: the file to upload - `data`: a JSON string with metadata, such as `indexing_technique`, `process_rule`, and optionally `doc_form` for hierarchical mode. Include the header: ``` Authorization: Bearer {api_key} ``` Example cURL: ```bash curl --location --request POST 'https://api.dify.ai/v1/datasets/{dataset_id}/document/create-by-file' \ --header 'Authorization: Bearer {api_key}' \ --form 'data="{\"indexing_technique\":\"high_quality\",\"process_rule\":{\"mode\":\"hierarchical\"},\"doc_form\":\"hierarchical_model\"}";type=text/plain' \ --form 'file=@"path/to/your/file.md"' ``` After upload, Dify will automatically process and parse the file in the knowledge base pipeline. For advanced pipeline features (like custom extractors), API support may be limited compared to the Web UI, but using a user-scoped API token and matching your `process_rule` to the pipeline config maximizes compatibility. For more details and examples, see [this issue](https://github.com/langgenius/dify/issues/15085) and [this discussion](https://github.com/langgenius/dify/issues/26649). If you use the Python SDK, the `KnowledgeBaseClient.upload_pipeline_file()` method is designed for this purpose [reference](https://github.com/langgenius/dify-docs/issues/486). For large files (>800KB), use a Python code node instead of the HTTP request node in Dify workflows [reference](https://github.com/langgenius/dify/issues/20307).  *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/897092cc-cfa2-4e0d-8d78-cd896c5e9dfa?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/897092cc-cfa2-4e0d-8d78-cd896c5e9dfa?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/897092cc-cfa2-4e0d-8d78-cd896c5e9dfa?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/897092cc-cfa2-4e0d-8d78-cd896c5e9dfa?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/897092cc-cfa2-4e0d-8d78-cd896c5e9dfa?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/897092cc-cfa2-4e0d-8d78-cd896c5e9dfa?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/897092cc-cfa2-4e0d-8d78-cd896c5e9dfa?feedback_type=other)</sup>  [![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot) [![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify-docs/issues/507)

yindo commented

2026-02-16 05:19:39 -05:00

@xiaohe123456 commented on GitHub (Oct 31, 2025):

@dosu When uploading files via API and parsing PDF documents, MinerU was not utilized. Why is that? How can PDF documents be parsed without using MinerU, and what is the accuracy rate in this case?

@xiaohe123456 commented on GitHub (Oct 31, 2025): @dosu When uploading files via API and parsing PDF documents, MinerU was not utilized. Why is that? How can PDF documents be parsed without using MinerU, and what is the accuracy rate in this case?

yindo commented

2026-02-16 05:19:39 -05:00

@dosubot[bot] commented on GitHub (Oct 31, 2025):

MinerU is not used when uploading and parsing PDF documents via the API because the API endpoint (/v1/datasets/{dataset_id}/document/create-by-file) defaults to Dify's built-in parsing logic and does not reliably trigger the Knowledge Pipeline or custom extractors like MinerU. This is a known limitation—currently, only uploads through the Web UI are guaranteed to use the full pipeline with custom extractors like MinerU. When you upload a PDF via the API, Dify uses its own PdfExtractor class, which parses PDFs using the pypdfium2 library to extract text from each page (source).

The accuracy of this default method is generally good for text-based PDFs, but it does not perform OCR, so it cannot extract text from scanned or image-only PDFs. Complex layouts, tables, and images may not be parsed accurately. For best results, use text-based PDFs and minimize complex formatting. If you need higher accuracy or support for scanned/image PDFs, you would need to process the files externally (for example, with MinerU or another OCR tool) and then upload the extracted text to Dify (reference).

To reply, just mention @dosu.

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

@dosubot[bot] commented on GitHub (Oct 31, 2025):  MinerU is not used when uploading and parsing PDF documents via the API because the API endpoint (`/v1/datasets/{dataset_id}/document/create-by-file`) defaults to Dify's built-in parsing logic and does not reliably trigger the Knowledge Pipeline or custom extractors like MinerU. This is a known limitation—currently, only uploads through the Web UI are guaranteed to use the full pipeline with custom extractors like MinerU. When you upload a PDF via the API, Dify uses its own `PdfExtractor` class, which parses PDFs using the `pypdfium2` library to extract text from each page ([source](https://github.com/langgenius/dify/blob/f7f6b4a8b0029dfd57cb7f3ef1e59f7cb5f09003/api/core/rag/extractor/pdf_extractor.py)). The accuracy of this default method is generally good for text-based PDFs, but it does not perform OCR, so it cannot extract text from scanned or image-only PDFs. Complex layouts, tables, and images may not be parsed accurately. For best results, use text-based PDFs and minimize complex formatting. If you need higher accuracy or support for scanned/image PDFs, you would need to process the files externally (for example, with MinerU or another OCR tool) and then upload the extracted text to Dify ([reference](https://github.com/langgenius/dify/issues/26649)).  *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/623c04e6-fd40-4637-8dcf-0bc7567feb03?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/623c04e6-fd40-4637-8dcf-0bc7567feb03?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/623c04e6-fd40-4637-8dcf-0bc7567feb03?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/623c04e6-fd40-4637-8dcf-0bc7567feb03?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/623c04e6-fd40-4637-8dcf-0bc7567feb03?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/623c04e6-fd40-4637-8dcf-0bc7567feb03?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/623c04e6-fd40-4637-8dcf-0bc7567feb03?feedback_type=other)</sup>  [![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot) [![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify-docs/issues/507)

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: langgenius/dify-docs#260