langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 12:05:30 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

How can I trigger the knowledge base pipeline via API? #20361

New Issue
Closed
opened 2026-02-21 20:07:01 -05:00 by yindo · 3 comments
yindo commented 2026-02-21 20:07:01 -05:00
Owner
Copy Link
Copy Source

Originally created by @KrisHe666 on GitHub (Nov 17, 2025).

Self Checks

  • I have read the Contributing Guide and Language Policy.
  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report, otherwise it will be closed.
  • 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

1.9.2

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

How can I trigger the knowledge base pipeline via API? Also, when I upload files on the interface, they are not directly stored in the knowledge base; I don't know why.

Image

✔️ Expected Behavior

1

❌ Actual Behavior

No response

Originally created by @KrisHe666 on GitHub (Nov 17, 2025). ### Self Checks - [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542). - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report, otherwise it will be closed. - [x] 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：） - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version 1.9.2 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce How can I trigger the knowledge base pipeline via API? Also, when I upload files on the interface, they are not directly stored in the knowledge base; I don't know why. ![Image](https://github.com/user-attachments/assets/e7e81707-2081-47bb-96db-fed817fd2ed9) ### ✔️ Expected Behavior 1 ### ❌ Actual Behavior _No response_
yindo added the 🙋‍♂️ question label 2026-02-21 20:07:01 -05:00
yindo closed this issue 2026-02-21 20:07:01 -05:00
yindo commented 2026-02-21 20:07:02 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Nov 17, 2025):

To trigger the knowledge base pipeline via API in Dify v1.9.2 (self-hosted), you need to follow a two-step process:

  1. Upload your file using the API endpoint:

    POST /v1/datasets/{dataset_id}/document/create_by_file

    This will save the file and create a document entry, which starts the document processing pipeline. However, this may not run custom workflow steps exactly as in the UI, and some pipeline features might not be triggered reliably via API alone. Make sure your dataset and process rules are correctly configured for the file to be indexed and stored in the knowledge base.
    API details

  2. (Optional for advanced workflows) If you need to trigger a custom pipeline or workflow node (for example, with custom extractors or aggregators), use:

    POST /datasets/{dataset_id}/pipeline/run

    Provide the required JSON body, including references to your uploaded file, the datasource type, and the start node ID. This endpoint lets you run the pipeline programmatically, but you must coordinate the file upload and pipeline trigger manually.
    Pipeline run endpoint

Why uploaded files may not appear in the knowledge base:
Files uploaded via the interface or API are only stored in the knowledge base if your workflow includes a node that handles indexing (like KnowledgeIndexNode). If your workflow only outputs or retrieves data (e.g., with DatasourceNode or KnowledgeRetrievalNode), nothing is persisted by default. Make sure your workflow explicitly includes and configures a storage/indexing node with the correct dataset and document IDs.
Workflow node logic

Troubleshooting:

  • If files get stuck in the pipeline queue, check your Celery worker configuration. The worker must listen to both pipeline and priority_pipeline queues.
  • There is a known bug in v1.9.2 where indexing tasks may not trigger for pipeline knowledge bases. As a workaround, you can manually trigger indexing via a Python script.
  • Ensure the FILES_URL environment variable is set correctly for file previews and downloads.
    Queue/config details

For more details, see the API Reference and External Knowledge API Guide.

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other  Chat with Dosu Join Discord Share on X

@dosubot[bot] commented on GitHub (Nov 17, 2025): <!-- Answer --> To trigger the knowledge base pipeline via API in Dify v1.9.2 (self-hosted), you need to follow a two-step process: 1. **Upload your file** using the API endpoint: ``` POST /v1/datasets/{dataset_id}/document/create_by_file ``` This will save the file and create a document entry, which starts the document processing pipeline. However, this may not run custom workflow steps exactly as in the UI, and some pipeline features might not be triggered reliably via API alone. Make sure your dataset and process rules are correctly configured for the file to be indexed and stored in the knowledge base. [API details](https://github.com/langgenius/dify/blob/4a89403566c85eff76a959bc62be76f1e8af5ce6/api/controllers/service_api/dataset/document.py#L65-L344) 2. **(Optional for advanced workflows)** If you need to trigger a custom pipeline or workflow node (for example, with custom extractors or aggregators), use: ``` POST /datasets/{dataset_id}/pipeline/run ``` Provide the required JSON body, including references to your uploaded file, the datasource type, and the start node ID. This endpoint lets you run the pipeline programmatically, but you must coordinate the file upload and pipeline trigger manually. [Pipeline run endpoint](https://github.com/langgenius/dify/blob/4a89403566c85eff76a959bc62be76f1e8af5ce6/api/controllers/service_api/dataset/rag_pipeline/rag_pipeline_workflow.py#L122-L246) **Why uploaded files may not appear in the knowledge base:** Files uploaded via the interface or API are only stored in the knowledge base if your workflow includes a node that handles indexing (like `KnowledgeIndexNode`). If your workflow only outputs or retrieves data (e.g., with `DatasourceNode` or `KnowledgeRetrievalNode`), nothing is persisted by default. Make sure your workflow explicitly includes and configures a storage/indexing node with the correct dataset and document IDs. [Workflow node logic](https://github.com/langgenius/dify/issues/27442) **Troubleshooting:** - If files get stuck in the pipeline queue, check your Celery worker configuration. The worker must listen to both `pipeline` and `priority_pipeline` queues. - There is a known bug in v1.9.2 where indexing tasks may not trigger for pipeline knowledge bases. As a workaround, you can manually trigger indexing via a Python script. - Ensure the `FILES_URL` environment variable is set correctly for file previews and downloads. [Queue/config details](https://github.com/langgenius/dify/discussions/28202) For more details, see the [API Reference](https://docs.dify.ai/api-reference/) and [External Knowledge API Guide](https://docs.dify.ai/guides/knowledge-base/external-knowledge-api-documentation). <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/4204b67a-662f-4638-876f-50f30cc8d1bf?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/4204b67a-662f-4638-876f-50f30cc8d1bf?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/4204b67a-662f-4638-876f-50f30cc8d1bf?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/4204b67a-662f-4638-876f-50f30cc8d1bf?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/4204b67a-662f-4638-876f-50f30cc8d1bf?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/4204b67a-662f-4638-876f-50f30cc8d1bf?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/4204b67a-662f-4638-876f-50f30cc8d1bf?feedback_type=other)</sup>&nbsp;&nbsp;[![Chat with Dosu](https://dosu.dev/dosu-chat-badge.svg)](https://app.dosu.dev/6088a4f4-073a-43f3-b3d4-4f06fb0c05a0/ask?utm_source=github)&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/28278)
yindo commented 2026-02-21 20:07:02 -05:00
Author
Owner
Copy Link
Copy Source

@keminar commented on GitHub (Dec 25, 2025):

@crazywoola
why close this issue.
I want to know how to request /datasets/{dataset_id}/pipeline/run too

when i request /datasets/5e96ee75-3fa8-41ad-a002-a8ae09283d50/pipeline/run
HTTP/1.1 404 Not Found

@keminar commented on GitHub (Dec 25, 2025): @crazywoola why close this issue. I want to know how to request /datasets/{dataset_id}/pipeline/run too when i request /datasets/5e96ee75-3fa8-41ad-a002-a8ae09283d50/pipeline/run HTTP/1.1 404 Not Found
yindo commented 2026-02-21 20:07:02 -05:00
Author
Owner
Copy Link
Copy Source

@keminar commented on GitHub (Dec 26, 2025):

@KrisHe666

Try using the patch here.

https://github.com/langgenius/dify/issues/30203

@keminar commented on GitHub (Dec 26, 2025): @KrisHe666 Try using the patch here. https://github.com/langgenius/dify/issues/30203
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🙋‍♂️ question
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#20361
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?