langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 03:55:25 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Uploaded file name in local machine doesn't match the uploaded api response #4042

New Issue
Closed
opened 2026-02-21 18:04:31 -05:00 by yindo · 5 comments
yindo commented 2026-02-21 18:04:31 -05:00
Owner
Copy Link
Copy Source

Originally created by @AlexSu1108 on GitHub (Jun 12, 2024).

Self Checks

  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit report (我已阅读并同意 Language Policy).
  • Please do not modify this template :) and fill in all the required fields.

Provide a description of requested docs changes

I uploaded a file using API: "/document/create_by_file"
I got the response:
{"document": {"id": "c6c3f4b5-4376-414c-a60a-f9433961f5aa", "position": 1, "data_source_type": "upload_file", "data_source_info": {"upload_file_id": "3d8e4bd4-47db-4973-8086-4eb61ca94b46"}, "data_source_detail_dict": {"upload_file": {"id": "3d8e4bd4-47db-4973-8086-4eb61ca94b46", "name": "49965.txt", "size": 2296, "extension": "txt", "mime_type": "text/plain", "created_by": "6e66b67f-011b-437d-beab-32064095cc1e", "created_at": 1718167685.283874}}, "dataset_process_rule_id": "a1e59d16-ac5e-4cf9-bf39-4f9b8628d03a", "name": "49965.txt", "created_from": "api", "created_by": "6e66b67f-011b-437d-beab-32064095cc1e", "created_at": 1718167685, "tokens": 0, "indexing_status": "waiting", "error": null, "enabled": true, "disabled_at": null, "disabled_by": null, "archived": false, "display_status": "queuing", "word_count": 0, "hit_count": 0, "doc_form": "text_model"}, "batch": "20240612044805748679"}

Apparently, I hope I can find the file with the name shown in the response (like: id, upload_file_id, etc).

However, when I dive into the local machine, what I found is like this:

[root@VM-1-3-tencentos bdf9f4ed-fcd1-4e34-84b7-49911994a556]# pwd
/root/dify/docker/volumes/app/storage/upload_files/bdf9f4ed-fcd1-4e34-84b7-49911994a556
[root@VM-1-3-tencentos bdf9f4ed-fcd1-4e34-84b7-49911994a556]# ls
5a24e50f-707c-41ad-ab94-394c4b11fa8a.txt abe59dec-8c23-444e-9938-f2eb1bc35024.txt

My questions are:

  1. I only uploaded one document, but two local files were found on the local machine. Why?
  2. The local file name doesn't match any ids shown in the response of "upload file API". Why?
  3. The meanings of ids shown in the response need to be cleared in the documentation.
Originally created by @AlexSu1108 on GitHub (Jun 12, 2024). ### Self Checks - [X] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [X] I confirm that I am using English to submit report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [X] Please do not modify this template :) and fill in all the required fields. ### Provide a description of requested docs changes I uploaded a file using API: "/document/create_by_file" I got the response: `{"document": {"id": "c6c3f4b5-4376-414c-a60a-f9433961f5aa", "position": 1, "data_source_type": "upload_file", "data_source_info": {"upload_file_id": "3d8e4bd4-47db-4973-8086-4eb61ca94b46"}, "data_source_detail_dict": {"upload_file": {"id": "3d8e4bd4-47db-4973-8086-4eb61ca94b46", "name": "49965.txt", "size": 2296, "extension": "txt", "mime_type": "text/plain", "created_by": "6e66b67f-011b-437d-beab-32064095cc1e", "created_at": 1718167685.283874}}, "dataset_process_rule_id": "a1e59d16-ac5e-4cf9-bf39-4f9b8628d03a", "name": "49965.txt", "created_from": "api", "created_by": "6e66b67f-011b-437d-beab-32064095cc1e", "created_at": 1718167685, "tokens": 0, "indexing_status": "waiting", "error": null, "enabled": true, "disabled_at": null, "disabled_by": null, "archived": false, "display_status": "queuing", "word_count": 0, "hit_count": 0, "doc_form": "text_model"}, "batch": "20240612044805748679"}` Apparently, I hope I can find the file with the name shown in the response (like: id, upload_file_id, etc). However, when I dive into the local machine, what I found is like this: [root@VM-1-3-tencentos bdf9f4ed-fcd1-4e34-84b7-49911994a556]# pwd /root/dify/docker/volumes/app/storage/upload_files/bdf9f4ed-fcd1-4e34-84b7-49911994a556 [root@VM-1-3-tencentos bdf9f4ed-fcd1-4e34-84b7-49911994a556]# ls 5a24e50f-707c-41ad-ab94-394c4b11fa8a.txt abe59dec-8c23-444e-9938-f2eb1bc35024.txt My questions are: 1. I only uploaded one document, but two local files were found on the local machine. Why? 2. The local file name doesn't match any ids shown in the response of "upload file API". Why? 3. The meanings of ids shown in the response need to be cleared in the documentation.
yindo added the 🐞 bug🙋‍♂️ question labels 2026-02-21 18:04:31 -05:00
yindo closed this issue 2026-02-21 18:04:31 -05:00
yindo commented 2026-02-21 18:04:32 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Jun 12, 2024):

  1. Why are there two local files after uploading a single document?

    The two local files with different names after uploading a single document likely represent different stages or aspects of the document processing. The document is processed in segments, and these segments are saved as separate files. The IDs in the API response represent unique identifiers for the documents and their statuses, which help track the document through different stages of processing [1][2].

  2. Why doesn't the local file name match any IDs shown in the API response?

    The local file names are generated by concatenating the base folder path (defined by the STORAGE_LOCAL_PATH configuration) with the provided filename. The naming convention ensures that the full path is constructed correctly by checking if the folder path ends with a '/' and appending the filename accordingly. This process does not necessarily use the IDs from the API response for naming the files [3].

  3. Clarification of the IDs in the API response:

    • document.id: Unique identifier for the document.
    • data_source_info.upload_file_id: Unique identifier for the uploaded file.
    • data_source_detail_dict.upload_file.id: Another unique identifier for the uploaded file, which might be used internally for tracking.
    • dataset_process_rule_id: Identifier for the processing rule applied to the dataset.

    These IDs help in tracking and managing the document and its processing status.

To continue talking to Dosu, mention @dosu.

@dosubot[bot] commented on GitHub (Jun 12, 2024): <!-- Answer --> 1. **Why are there two local files after uploading a single document?** The two local files with different names after uploading a single document likely represent different stages or aspects of the document processing. The document is processed in segments, and these segments are saved as separate files. The IDs in the API response represent unique identifiers for the documents and their statuses, which help track the document through different stages of processing <sup>[[1]](https://github.com/langgenius/dify/blob/main/api/fields/document_fields.py)</sup><sup>[[2]](https://github.com/langgenius/dify/blob/main/api/controllers/service_api/dataset/document.py)</sup>. 2. **Why doesn't the local file name match any IDs shown in the API response?** The local file names are generated by concatenating the base folder path (defined by the `STORAGE_LOCAL_PATH` configuration) with the provided filename. The naming convention ensures that the full path is constructed correctly by checking if the folder path ends with a '/' and appending the filename accordingly. This process does not necessarily use the IDs from the API response for naming the files <sup>[[3]](https://github.com/langgenius/dify/blob/main/api/extensions/storage/local_storage.py)</sup>. 3. **Clarification of the IDs in the API response:** - `document.id`: Unique identifier for the document. - `data_source_info.upload_file_id`: Unique identifier for the uploaded file. - `data_source_detail_dict.upload_file.id`: Another unique identifier for the uploaded file, which might be used internally for tracking. - `dataset_process_rule_id`: Identifier for the processing rule applied to the dataset. These IDs help in tracking and managing the document and its processing status. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
yindo commented 2026-02-21 18:04:32 -05:00
Author
Owner
Copy Link
Copy Source

@AlexSu1108 commented on GitHub (Jun 12, 2024):

  1. Why are there two local files after uploading a single document?
    The two local files with different names after uploading a single document likely represent different stages or aspects of the document processing. The document is processed in segments, and these segments are saved as separate files. The IDs in the API response represent unique identifiers for the documents and their statuses, which help track the document through different stages of processing [1][2].

  2. Why doesn't the local file name match any IDs shown in the API response?
    The local file names are generated by concatenating the base folder path (defined by the STORAGE_LOCAL_PATH configuration) with the provided filename. The naming convention ensures that the full path is constructed correctly by checking if the folder path ends with a '/' and appending the filename accordingly. This process does not necessarily use the IDs from the API response for naming the files [3].

  3. Clarification of the IDs in the API response:

    • document.id: Unique identifier for the document.
    • data_source_info.upload_file_id: Unique identifier for the uploaded file.
    • data_source_detail_dict.upload_file.id: Another unique identifier for the uploaded file, which might be used internally for tracking.
    • dataset_process_rule_id: Identifier for the processing rule applied to the dataset.

    These IDs help in tracking and managing the document and its processing status.

To continue talking to Dosu, mention @dosu.

Thanks for your reply. I'm wondering, is that possible for me to download the uploaded file in any manner?

@AlexSu1108 commented on GitHub (Jun 12, 2024): > 1. **Why are there two local files after uploading a single document?** > The two local files with different names after uploading a single document likely represent different stages or aspects of the document processing. The document is processed in segments, and these segments are saved as separate files. The IDs in the API response represent unique identifiers for the documents and their statuses, which help track the document through different stages of processing [[1]](https://github.com/langgenius/dify/blob/main/api/fields/document_fields.py)[[2]](https://github.com/langgenius/dify/blob/main/api/controllers/service_api/dataset/document.py). > 2. **Why doesn't the local file name match any IDs shown in the API response?** > The local file names are generated by concatenating the base folder path (defined by the `STORAGE_LOCAL_PATH` configuration) with the provided filename. The naming convention ensures that the full path is constructed correctly by checking if the folder path ends with a '/' and appending the filename accordingly. This process does not necessarily use the IDs from the API response for naming the files [[3]](https://github.com/langgenius/dify/blob/main/api/extensions/storage/local_storage.py). > 3. **Clarification of the IDs in the API response:** > > * `document.id`: Unique identifier for the document. > * `data_source_info.upload_file_id`: Unique identifier for the uploaded file. > * `data_source_detail_dict.upload_file.id`: Another unique identifier for the uploaded file, which might be used internally for tracking. > * `dataset_process_rule_id`: Identifier for the processing rule applied to the dataset. > > These IDs help in tracking and managing the document and its processing status. > > _To continue talking to [Dosu](https://dosu.dev), mention @dosu._ Thanks for your reply. I'm wondering, is that possible for me to download the uploaded file in any manner?
yindo commented 2026-02-21 18:04:32 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Jun 12, 2024):

Yes, you can download the uploaded file. The LocalStorage class in Dify provides methods to handle file operations, including downloading files. You can use the download method to retrieve the file. Here is a relevant snippet from the LocalStorage class:

class LocalStorage(BaseStorage):
    # Other methods...

    def download(self, key, file_path):
        with open(os.path.join(os.getcwd(), key), "rb") as f:
            data = f.read()
        with open(file_path, "wb") as f:
            f.write(data)

To download the file, you need to know the key (file path) of the uploaded file. You can then specify the destination path where you want to save the downloaded file. Here is an example of how you might use this method:

storage = LocalStorage()
key = "path/to/uploaded/file.txt"  # Replace with the actual key
destination_path = "path/to/save/downloaded/file.txt"
storage.download(key, destination_path)

This will read the file from the specified key and write it to the destination path on your local machine [1].

To continue talking to Dosu, mention @dosu.

@dosubot[bot] commented on GitHub (Jun 12, 2024): <!-- Answer --> Yes, you can download the uploaded file. The `LocalStorage` class in Dify provides methods to handle file operations, including downloading files. You can use the `download` method to retrieve the file. Here is a relevant snippet from the `LocalStorage` class: ```python class LocalStorage(BaseStorage): # Other methods... def download(self, key, file_path): with open(os.path.join(os.getcwd(), key), "rb") as f: data = f.read() with open(file_path, "wb") as f: f.write(data) ``` To download the file, you need to know the key (file path) of the uploaded file. You can then specify the destination path where you want to save the downloaded file. Here is an example of how you might use this method: ```python storage = LocalStorage() key = "path/to/uploaded/file.txt" # Replace with the actual key destination_path = "path/to/save/downloaded/file.txt" storage.download(key, destination_path) ``` This will read the file from the specified key and write it to the destination path on your local machine <sup>[[1]](https://github.com/langgenius/dify/blob/main/api/extensions/storage/local_storage.py)</sup>. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
yindo commented 2026-02-21 18:04:32 -05:00
Author
Owner
Copy Link
Copy Source

@AlexSu1108 commented on GitHub (Jun 12, 2024):

Yes, you can download the uploaded file. The LocalStorage class in Dify provides methods to handle file operations, including downloading files. You can use the download method to retrieve the file. Here is a relevant snippet from the LocalStorage class:

class LocalStorage(BaseStorage):
    # Other methods...

    def download(self, key, file_path):
        with open(os.path.join(os.getcwd(), key), "rb") as f:
            data = f.read()
        with open(file_path, "wb") as f:
            f.write(data)

To download the file, you need to know the key (file path) of the uploaded file. You can then specify the destination path where you want to save the downloaded file. Here is an example of how you might use this method:

storage = LocalStorage()
key = "path/to/uploaded/file.txt"  # Replace with the actual key
destination_path = "path/to/save/downloaded/file.txt"
storage.download(key, destination_path)

This will read the file from the specified key and write it to the destination path on your local machine [1].

To continue talking to Dosu, mention @dosu.

How can I get the mapping or the relationship among document id, local file name, upload file id and upload file name?

@AlexSu1108 commented on GitHub (Jun 12, 2024): > Yes, you can download the uploaded file. The `LocalStorage` class in Dify provides methods to handle file operations, including downloading files. You can use the `download` method to retrieve the file. Here is a relevant snippet from the `LocalStorage` class: > > ```python > class LocalStorage(BaseStorage): > # Other methods... > > def download(self, key, file_path): > with open(os.path.join(os.getcwd(), key), "rb") as f: > data = f.read() > with open(file_path, "wb") as f: > f.write(data) > ``` > > To download the file, you need to know the key (file path) of the uploaded file. You can then specify the destination path where you want to save the downloaded file. Here is an example of how you might use this method: > > ```python > storage = LocalStorage() > key = "path/to/uploaded/file.txt" # Replace with the actual key > destination_path = "path/to/save/downloaded/file.txt" > storage.download(key, destination_path) > ``` > > This will read the file from the specified key and write it to the destination path on your local machine [[1]](https://github.com/langgenius/dify/blob/main/api/extensions/storage/local_storage.py). > > _To continue talking to [Dosu](https://dosu.dev), mention @dosu._ How can I get the mapping or the relationship among document id, local file name, upload file id and upload file name?
yindo commented 2026-02-21 18:04:32 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Jun 12, 2024):

To get the mapping or relationship among document.id, local file name, upload_file_id, and upload_file.name, you can query the UploadFile table in the database. Here’s how these elements are related:

  1. Local File Name: This is generated as a UUID during the upload process.
  2. upload_file_id: This is the primary key of the UploadFile entry in the database.
  3. upload_file.name: This is the original filename of the uploaded file.

To retrieve the mapping, you can query the UploadFile table using the upload_file_id to get the key (which includes the local file name) and the name (original filename).

Here is an example of how you can use the FileService class to get this information:

from services.file_service import FileService

# Assuming you have the upload_file_id
upload_file_id = 'your_upload_file_id_here'

# Retrieve the upload file details
upload_file = FileService.get_upload_file(upload_file_id)
local_file_name = upload_file.key  # This is the local file name (UUID)
original_file_name = upload_file.name  # This is the original file name

print(f"Local File Name: {local_file_name}")
print(f"Original File Name: {original_file_name}")

This code will help you map the upload_file_id to the local file name and the original file name [1].

To continue talking to Dosu, mention @dosu.

@dosubot[bot] commented on GitHub (Jun 12, 2024): <!-- Answer --> To get the mapping or relationship among `document.id`, local file name, `upload_file_id`, and `upload_file.name`, you can query the `UploadFile` table in the database. Here’s how these elements are related: 1. **Local File Name**: This is generated as a UUID during the upload process. 2. **`upload_file_id`**: This is the primary key of the `UploadFile` entry in the database. 3. **`upload_file.name`**: This is the original filename of the uploaded file. To retrieve the mapping, you can query the `UploadFile` table using the `upload_file_id` to get the `key` (which includes the local file name) and the `name` (original filename). Here is an example of how you can use the `FileService` class to get this information: ```python from services.file_service import FileService # Assuming you have the upload_file_id upload_file_id = 'your_upload_file_id_here' # Retrieve the upload file details upload_file = FileService.get_upload_file(upload_file_id) local_file_name = upload_file.key # This is the local file name (UUID) original_file_name = upload_file.name # This is the original file name print(f"Local File Name: {local_file_name}") print(f"Original File Name: {original_file_name}") ``` This code will help you map the `upload_file_id` to the local file name and the original file name <sup>[[1]](https://github.com/langgenius/dify/blob/main/api/services/file_service.py)</sup>. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug 🙋‍♂️ question
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#4042
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?