langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 12:05:30 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Unable to Rebuild Document Index After Dataset Migration #20122

New Issue
Closed
opened 2026-02-21 20:05:56 -05:00 by yindo · 1 comment
yindo commented 2026-02-21 20:05:56 -05:00
Owner
Copy Link
Copy Source

Originally created by @Gaopeng-Bai on GitHub (Nov 4, 2025).

Self Checks

  • I have read the Contributing Guide and Language Policy.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report, otherwise it will be closed.
  • Please do not modify this template :) and fill in all the required fields.

1. Is this request related to a challenge you're experiencing? Tell me about your story.

After migrating datasets and documents between two Dify instances using the API (append_segments method),
the migrated documents appear in the target dataset, but indexing never starts or completes.
The document remains in the "waiting" or "not indexed" state, with no way to manually trigger re-indexing.

Steps to Reproduce:

Create a source dataset (Dataset A) with one or more documents.

Export or read all segments from Dataset A using the /datasets/{dataset_id}/documents/{document_id}/segments API.

In Dataset B, create an empty document via /datasets/{dataset_id}/document/create-by-text with an empty string.

Append all segments to the new document in Dataset B using:

POST /v1/datasets/{dataset_id}/documents/{document_id}/segments

Check the document status or try to search within Dataset B.
The new document appears in the dataset, but:

It has no sub-segments in the UI.
The indexing status remains "not indexed".

2. Additional context or comments

There is no API endpoint to manually trigger reindexing.

The internal task batch_create_segment_to_index_task exists in the source code,
but it’s not exposed as a REST endpoint.

3. Can you help us with this feature?

  • I am interested in contributing to this feature.
Originally created by @Gaopeng-Bai on GitHub (Nov 4, 2025). ### Self Checks - [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report, otherwise it will be closed. - [x] Please do not modify this template :) and fill in all the required fields. ### 1. Is this request related to a challenge you're experiencing? Tell me about your story. After migrating datasets and documents between two Dify instances using the API (append_segments method), the migrated documents appear in the target dataset, but indexing never starts or completes. The document remains in the "waiting" or "not indexed" state, with no way to manually trigger re-indexing. ### Steps to Reproduce: Create a source dataset (Dataset A) with one or more documents. Export or read all segments from Dataset A using the /datasets/{dataset_id}/documents/{document_id}/segments API. In Dataset B, create an empty document via /datasets/{dataset_id}/document/create-by-text with an empty string. Append all segments to the new document in Dataset B using: `POST /v1/datasets/{dataset_id}/documents/{document_id}/segments` Check the document status or try to search within Dataset B. The new document appears in the dataset, but: > It has no sub-segments in the UI. > The indexing status remains "not indexed". ### 2. Additional context or comments There is no API endpoint to manually trigger reindexing. The internal task batch_create_segment_to_index_task exists in the source code, but it’s not exposed as a REST endpoint. ### 3. Can you help us with this feature? - [ ] I am interested in contributing to this feature.
yindo added the 💪 enhancement label 2026-02-21 20:05:56 -05:00
yindo closed this issue 2026-02-21 20:05:56 -05:00
yindo commented 2026-02-21 20:05:56 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Nov 4, 2025):

There’s currently no public API or UI button in Dify to manually trigger reindexing for documents that remain in “waiting” or “not indexed” status after migration via the append_segments API. The internal task (batch_create_segment_to_index_task) is not exposed as a REST endpoint, and recent code changes have only refactored its internal usage, not made it accessible externally (source).

Workarounds depend on your deployment:

  • For self-hosted setups, you can manually trigger indexing for stuck documents by running Python scripts inside your Docker container, such as retry_document_indexing_task or recover_document_indexing_task (example).
  • You can also use the POST /datasets/<dataset_id>/retry endpoint to enqueue reindexing for one or more documents by passing their IDs in the request body (details). This is the closest available method to programmatically retry indexing.
  • Sometimes, toggling the document’s enabled/disabled status or updating its content via the update_by_text API will reset its indexing status and trigger reindexing, provided the document is not archived or in error state (reference).
  • For managed/cloud users without backend access, options are limited to UI actions (edit, save, enable/disable) or using the retry endpoint if available.

If you’re seeing persistent issues, make sure your background worker (Celery) is running and healthy, as indexing tasks depend on it (troubleshooting). There’s no “full re-index” button, but bulk reindexing is possible by scripting calls to the retry endpoint for all documents in a dataset.

If you need more details on scripting or backend task invocation, let me know your deployment type and access level.

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other  Chat with Dosu Join Discord Share on X

@dosubot[bot] commented on GitHub (Nov 4, 2025): <!-- Answer --> There’s currently no public API or UI button in Dify to manually trigger reindexing for documents that remain in “waiting” or “not indexed” status after migration via the append_segments API. The internal task (`batch_create_segment_to_index_task`) is not exposed as a REST endpoint, and recent code changes have only refactored its internal usage, not made it accessible externally ([source](https://github.com/langgenius/dify/pull/26795)). Workarounds depend on your deployment: - For self-hosted setups, you can manually trigger indexing for stuck documents by running Python scripts inside your Docker container, such as `retry_document_indexing_task` or `recover_document_indexing_task` ([example](https://github.com/langgenius/dify/issues/27763)). - You can also use the `POST /datasets/<dataset_id>/retry` endpoint to enqueue reindexing for one or more documents by passing their IDs in the request body ([details](https://github.com/langgenius/dify/discussions/26691)). This is the closest available method to programmatically retry indexing. - Sometimes, toggling the document’s enabled/disabled status or updating its content via the `update_by_text` API will reset its indexing status and trigger reindexing, provided the document is not archived or in error state ([reference](https://github.com/langgenius/dify/issues/26698)). - For managed/cloud users without backend access, options are limited to UI actions (edit, save, enable/disable) or using the retry endpoint if available. If you’re seeing persistent issues, make sure your background worker (Celery) is running and healthy, as indexing tasks depend on it ([troubleshooting](https://github.com/langgenius/dify/issues/12876)). There’s no “full re-index” button, but bulk reindexing is possible by scripting calls to the retry endpoint for all documents in a dataset. If you need more details on scripting or backend task invocation, let me know your deployment type and access level. <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/2bab1895-e91b-4796-906c-6d1ab103d78c?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/2bab1895-e91b-4796-906c-6d1ab103d78c?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/2bab1895-e91b-4796-906c-6d1ab103d78c?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/2bab1895-e91b-4796-906c-6d1ab103d78c?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/2bab1895-e91b-4796-906c-6d1ab103d78c?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/2bab1895-e91b-4796-906c-6d1ab103d78c?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/2bab1895-e91b-4796-906c-6d1ab103d78c?feedback_type=other)</sup>&nbsp;&nbsp;[![Chat with Dosu](https://dosu.dev/dosu-chat-badge.svg)](https://app.dosu.dev/6088a4f4-073a-43f3-b3d4-4f06fb0c05a0/ask?utm_source=github)&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/27816)
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 💪 enhancement
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#20122
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?