langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 12:05:30 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

How to Integrate Knowledge Base Pipeline via API #21475

New Issue
Closed
opened 2026-02-21 20:12:45 -05:00 by yindo · 1 comment
yindo commented 2026-02-21 20:12:45 -05:00
Owner
Copy Link
Copy Source

Originally created by @wuyuanhui on GitHub (Jan 6, 2026).

Self Checks

  • I have read the Contributing Guide and Language Policy.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report, otherwise it will be closed.
  • Please do not modify this template :) and fill in all the required fields.

1. Is this request related to a challenge you're experiencing? Tell me about your story.

I'm using the knowledge base pipeline feature in version 1.10.1 and would like to inquire about API integration. I couldn't find relevant documentation about knowledge base pipeline integration in the API documentation. Could you please clarify:

Are there API endpoints available for programmatically triggering/managing knowledge base pipelines?
Besides manual file uploads through the UI, what alternative integration methods are supported (e.g., direct API file submissions, webhook triggers, or SDK implementations)?
If API integration is available, could you provide documentation or examples of request/response formats and authentication requirements?
This information would be very helpful for implementing automated workflows. Thank you for your support!

Environment:

Version: 1.10.1
Current Usage: Manual file upload via UI

2. Additional context or comments

No response

3. Can you help us with this feature?

  • I am interested in contributing to this feature.
Originally created by @wuyuanhui on GitHub (Jan 6, 2026). ### Self Checks - [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report, otherwise it will be closed. - [x] Please do not modify this template :) and fill in all the required fields. ### 1. Is this request related to a challenge you're experiencing? Tell me about your story. I'm using the knowledge base pipeline feature in version 1.10.1 and would like to inquire about API integration. I couldn't find relevant documentation about knowledge base pipeline integration in the API documentation. Could you please clarify: Are there API endpoints available for programmatically triggering/managing knowledge base pipelines? Besides manual file uploads through the UI, what alternative integration methods are supported (e.g., direct API file submissions, webhook triggers, or SDK implementations)? If API integration is available, could you provide documentation or examples of request/response formats and authentication requirements? This information would be very helpful for implementing automated workflows. Thank you for your support! Environment: Version: 1.10.1 Current Usage: Manual file upload via UI ### 2. Additional context or comments _No response_ ### 3. Can you help us with this feature? - [ ] I am interested in contributing to this feature.
yindo added the 🙋‍♂️ question label 2026-02-21 20:12:45 -05:00
yindo closed this issue 2026-02-21 20:12:45 -05:00
yindo commented 2026-02-21 20:12:45 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Jan 6, 2026):

Hi @wuyuanhui! I'm Dosu and I’m helping the dify team.

Dify v1.10.1 supports API-based automation for knowledge base pipelines, but it requires a two-step process: upload the file, then trigger the pipeline run. There are no official webhook triggers or SDKs, and the process is not fully documented, but here’s how it works:

1. Upload a file to the pipeline:

Use POST /datasets/pipeline/file-upload with multipart/form-data. You must include your dataset-specific API key as a Bearer token in the Authorization header. Example cURL:

curl -X POST 'https://your-dify-server/v1/datasets/pipeline/file-upload' \
  -H 'Authorization: Bearer {dataset_api_key}' \
  -F 'file=@/path/to/your/file.pdf'

The response will include file metadata (id, name, size, etc.) if successful
API controller reference.

2. Trigger the pipeline run:

Use POST /datasets/{dataset_id}/pipeline/run with a JSON body referencing the uploaded file and pipeline configuration. Example payload:

{
  "inputs": {},
  "start_node_id": "your_start_node_id",
  "datasource_type": "local_file",
  "datasource_info_list": [
    {
      "related_id": "file_id_from_upload",
      "name": "your_file.pdf",
      "size": 123456,
      "extension": "pdf",
      "mime_type": "application/pdf",
      "url": "",
      "transfer_method": "local_file",
      "credential_id": ""
    }
  ],
  "is_published": true,
  "response_mode": "blocking"
}

Example cURL:

curl -X POST 'https://your-dify-server/v1/datasets/{dataset_id}/pipeline/run' \
  -H 'Authorization: Bearer {dataset_api_key}' \
  -H 'Content-Type: application/json' \
  -d @payload.json

Authentication:
You must use a dataset-specific API key (not a universal key) as a Bearer token. Generate this key in the Dify UI for each knowledge base you want to automate
API key info.

Limitations and caveats:

  • Only certain file types (pdf, docx, txt) are supported; images are not accepted
  • There is no single API call that both uploads and triggers the pipeline—you must do both steps
  • The /pipeline/run endpoint is experimental and may not work reliably in all environments; UI uploads are more robust for custom pipelines
  • Files uploaded via API may not trigger full LLM analysis or hierarchical segmentation as in the UI; they may be processed with simple segmentation only
  • Batch document upload is supported, but bulk creation of knowledge bases is not—script repeated API calls as a workaround
  • Webhook triggers and SDKs are not officially supported in v1.10.1
    API usage and caveats
    Discussion of limitations

Best practices:

  • Always check your dataset’s runtime_mode to ensure it’s in pipeline mode
  • If you encounter errors or missing features, check for updates in the official API docs and community discussions, as the API is evolving

Let me know if you need more detailed examples or troubleshooting tips for your specific workflow.

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other

Chat with Dosu Open in Cursor Join Discord Share on X

@dosubot[bot] commented on GitHub (Jan 6, 2026): <!-- Greeting --> Hi @wuyuanhui! I'm [Dosu](https://go.dosu.dev/dosubot) and I’m helping the dify team. <!-- Answer --> Dify v1.10.1 supports API-based automation for knowledge base pipelines, but it requires a two-step process: upload the file, then trigger the pipeline run. There are no official webhook triggers or SDKs, and the process is not fully documented, but here’s how it works: **1. Upload a file to the pipeline:** Use `POST /datasets/pipeline/file-upload` with `multipart/form-data`. You must include your dataset-specific API key as a Bearer token in the `Authorization` header. Example cURL: ```bash curl -X POST 'https://your-dify-server/v1/datasets/pipeline/file-upload' \ -H 'Authorization: Bearer {dataset_api_key}' \ -F 'file=@/path/to/your/file.pdf' ``` The response will include file metadata (id, name, size, etc.) if successful [API controller reference](https://github.com/langgenius/dify/blob/64bfcbc4a98b612244ee96ac1ac1c62874fd3191/api/controllers/service_api/dataset/rag_pipeline/rag_pipeline_workflow.py#L33-L243). **2. Trigger the pipeline run:** Use `POST /datasets/{dataset_id}/pipeline/run` with a JSON body referencing the uploaded file and pipeline configuration. Example payload: ```json { "inputs": {}, "start_node_id": "your_start_node_id", "datasource_type": "local_file", "datasource_info_list": [ { "related_id": "file_id_from_upload", "name": "your_file.pdf", "size": 123456, "extension": "pdf", "mime_type": "application/pdf", "url": "", "transfer_method": "local_file", "credential_id": "" } ], "is_published": true, "response_mode": "blocking" } ``` Example cURL: ```bash curl -X POST 'https://your-dify-server/v1/datasets/{dataset_id}/pipeline/run' \ -H 'Authorization: Bearer {dataset_api_key}' \ -H 'Content-Type: application/json' \ -d @payload.json ``` **Authentication:** You must use a dataset-specific API key (not a universal key) as a Bearer token. Generate this key in the Dify UI for each knowledge base you want to automate [API key info](https://github.com/langgenius/dify/issues/29154). **Limitations and caveats:** - Only certain file types (pdf, docx, txt) are supported; images are not accepted - There is no single API call that both uploads and triggers the pipeline—you must do both steps - The `/pipeline/run` endpoint is experimental and may not work reliably in all environments; UI uploads are more robust for custom pipelines - Files uploaded via API may not trigger full LLM analysis or hierarchical segmentation as in the UI; they may be processed with simple segmentation only - Batch document upload is supported, but bulk creation of knowledge bases is not—script repeated API calls as a workaround - Webhook triggers and SDKs are not officially supported in v1.10.1 [API usage and caveats](https://github.com/langgenius/dify/issues/30203) [Discussion of limitations](https://github.com/langgenius/dify/issues/27464) **Best practices:** - Always check your dataset’s `runtime_mode` to ensure it’s in pipeline mode - If you encounter errors or missing features, check for updates in the [official API docs](https://docs.dify.ai/api-reference/) and community discussions, as the API is evolving Let me know if you need more detailed examples or troubleshooting tips for your specific workflow. <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/eee72502-8396-4291-9ab7-7bbdab012484?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/eee72502-8396-4291-9ab7-7bbdab012484?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/eee72502-8396-4291-9ab7-7bbdab012484?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/eee72502-8396-4291-9ab7-7bbdab012484?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/eee72502-8396-4291-9ab7-7bbdab012484?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/eee72502-8396-4291-9ab7-7bbdab012484?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/eee72502-8396-4291-9ab7-7bbdab012484?feedback_type=other)</sup> [![Chat with Dosu](https://dosu.dev/dosu-chat-badge.svg)](https://app.dosu.dev/6088a4f4-073a-43f3-b3d4-4f06fb0c05a0/ask?utm_source=github)&nbsp;[![Open in Cursor](https://dosu.dev/dosu-in-cursor.svg)](https://cursor.com/link/prompt?text=Dify%20v1.10.1%20supports%20API-based%20automation%20for%20knowledge%20base%20pipelines%2C%20but%20it%20requires%20a%20two-step%20process%3A%20upload%20the%20file%2C%20then%20trigger%20the%20pipeline%20run.%20There%20are%20no%20official%20webhook%20triggers%20or%20SDKs%2C%20and%20the%20process%20is%20not%20fully%20documented%2C%20but%20here%E2%80%99s%20how%20it%20works%3A%0A%0A%2A%2A1.%20Upload%20a%20file%20to%20the%20pipeline%3A%2A%2A%0A%0AUse%20%60POST%20/datasets/pipeline/file-upload%60%20with%20%60multipart/form-data%60.%20You%20must%20include%20your%20dataset-specific%20API%20key%20as%20a%20Bearer%20token%20in%20the%20%60Authorization%60%20header.%20Example%20cURL%3A%0A%0A%60%60%60bash%0Acurl%20-X%20POST%20%27https%3A//your-dify-server/v1/datasets/pipeline/file-upload%27%20%5C%0A%20%20-H%20%27Authorization%3A%20Bearer%20%7Bdataset_api_key%7D%27%20%5C%0A%20%20-F%20%27file%3D%40/path/to/your/file.pdf%27%0A%60%60%60%0A%0AThe%20response%20will%20include%20file%20metadata%20%28id%2C%20name%2C%20size%2C%20etc.%29%20if%20successful%20%20%0A%5BAPI%20controller%20reference%5D%28https%3A//github.com/langgenius/dify/blob/64bfcbc4a98b612244ee96ac1ac1c62874fd3191/api/controllers/service_api/dataset/rag_pipeline/rag_pipeline_workflow.py%23L33-L243%29.%0A%0A%2A%2A2.%20Trigger%20the%20pipeline%20run%3A%2A%2A%0A%0AUse%20%60POST%20/datasets/%7Bdataset_id%7D/pipeline/run%60%20with%20a%20JSON%20body%20referencing%20the%20uploaded%20file%20and%20pipeline%20configuration.%20Example%20payload%3A%0A%0A%60%60%60json%0A%7B%0A%20%20%22inputs%22%3A%20%7B%7D%2C%0A%20%20%22start_node_id%22%3A%20%22your_start_node_id%22%2C%0A%20%20%22datasource_type%22%3A%20%22local_file%22%2C%0A%20%20%22datasource_info_list%22%3A%20%5B%0A%20%20%20%20%7B%0A%20%20%20%20%20%20%22related_id%22%3A%20%22file_id_from_upload%22%2C%0A%20%20%20%20%20%20%22name%22%3A%20%22your_file.pdf%22%2C%0A%20%20%20%20%20%20%22size%22%3A%20123456%2C%0A%20%20%20%20%20%20%22extension%22%3A%20%22pdf%22%2C%0A%20%20%20%20%20%20%22mime_type%22%3A%20%22application/pdf%22%2C%0A%20%20%20%20%20%20%22url%22%3A%20%22%22%2C%0A%20%20%20%20%20%20%22transfer_method%22%3A%20%22local_file%22%2C%0A%20%20%20%20%20%20%22credential_id%22%3A%20%22%22%0A%20%20%20%20%7D%0A%20%20%5D%2C%0A%20%20%22is_published%22%3A%20true%2C%0A%20%20%22response_mode%22%3A%20%22blocking%22%0A%7D%0A%60%60%60%0A%0AExample%20cURL%3A%0A%0A%60%60%60bash%0Acurl%20-X%20POST%20%27https%3A//your-dify-server/v1/datasets/%7Bdataset_id%7D/pipeline/run%27%20%5C%0A%20%20-H%20%27Authorization%3A%20Bearer%20%7Bdataset_api_key%7D%27%20%5C%0A%20%20-H%20%27Content-Type%3A%20application/json%27%20%5C%0A%20%20-d%20%40payload.json%0A%60%60%60%0A%0A%2A%2AAuthentication%3A%2A%2A%20%20%0AYou%20must%20use%20a%20dataset-specific%20API%20key%20%28not%20a%20universal%20key%29%20as%20a%20Bearer%20token.%20Generate%20this%20key%20in%20the%20Dify%20UI%20for%20each%20knowledge%20base%20you%20want%20to%20automate%20%20%0A%5BAPI%20key%20info%5D%28https%3A//github.com/langgenius/dify/issues/29154%29.%0A%0A%2A%2ALimitations%20and%20caveats%3A%2A%2A%20%20%0A-%20Only%20certain%20file%20types%20%28pdf%2C%20docx%2C%20txt%29%20are%20supported%3B%20images%20are%20not%20accepted%20%20%0A-%20There%20is%20no%20single%20API%20call%20that%20both%20uploads%20and%20triggers%20the%20pipeline%E2%80%94you%20must%20do%20both%20steps%20%20%0A-%20The%20%60/pipeline/run%60%20endpoint%20is%20experimental%20and%20may%20not%20work%20reliably%20in%20all%20environments%3B%20UI%20uploads%20are%20more%20robust%20for%20custom%20pipelines%20%20%0A-%20Files%20uploaded%20via%20API%20may%20not%20trigger%20full%20LLM%20analysis%20or%20hierarchical%20segmentation%20as%20in%20the%20UI%3B%20they%20may%20be%20processed%20with%20simple%20segmentation%20only%20%20%0A-%20Batch%20document%20upload%20is%20supported%2C%20but%20bulk%20creation%20of%20knowledge%20bases%20is%20not%E2%80%94script%20repeated%20API%20calls%20as%20a%20workaround%20%20%0A-%20Webhook%20triggers%20and%20SDKs%20are%20not%20officially%20supported%20in%20v1.10.1%20%20%0A%5BAPI%20usage%20and%20caveats%5D%28https%3A//github.com/langgenius/dify/issues/30203%29%20%20%0A%5BDiscussion%20of%20limitations%5D%28https%3A//github.com/langgenius/dify/issues/27464%29%0A%0A%2A%2ABest%20practices%3A%2A%2A%20%20%0A-%20Always%20check%20your%20dataset%E2%80%99s%20%60runtime_mode%60%20to%20ensure%20it%E2%80%99s%20in%20pipeline%20mode%20%20%0A-%20If%20you%20encounter%20errors%20or%20missing%20features%2C%20check%20for%20updates%20in%20the%20%5Bofficial%20API%20docs%5D%28https%3A//docs.dify.ai/api-reference/%29%20and%20community%20discussions%2C%20as%20the%20API%20is%20evolving%0A%0ALet%20me%20know%20if%20you%20need%20more%20detailed%20examples%20or%20troubleshooting%20tips%20for%20your%20specific%20workflow.)&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/30645)
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🙋‍♂️ question
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#21475
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?