langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-22 12:36:14 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Why Not Support Multi-File Extraction in HttpRequestNode.extract_files #13231

New Issue
Closed
opened 2026-02-21 19:11:11 -05:00 by yindo · 1 comment
yindo commented 2026-02-21 19:11:11 -05:00
Owner
Copy Link
Copy Source

Originally created by @signhuwei on GitHub (Apr 22, 2025).

Self Checks

  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:)
  • Please do not modify this template :) and fill in all the required fields.

1. Is this request related to a challenge you're experiencing? Tell me about your story.

Description:

Currently, the extract_files method in HttpRequestNode processes HTTP responses as a single file (even for multipart responses). This limits scenarios where the third-party API returns multiple files (e.g., batch image crops or multipart/form-data).

Proposed Changes:

  1. Detect multipart responses:
    Check Content-Type for multipart/ and parse each part using requests_toolbelt.MultipartDecoder.

  2. Handle each file individually:
    For each part with filename in Content-Disposition, create a separate File object via ToolFileManager.

  3. Fallback to current logic:
    Retain existing behavior for non-multipart responses (single file).

Example Modified Code:

from requests_toolbelt.multipart import decoder

def extract_files(self, url: str, response: Response) -> list[File]:
    if 'multipart/' in response.headers.get('Content-Type', ''):
        multipart_data = decoder.MultipartDecoder.from_response(response)
        for part in multipart_data.parts:
            if b'filename=' in part.headers.get(b'Content-Disposition', b''):
                # Create File for each part
                ...
    elif response.is_file:
        # Existing single-file logic
        ...

Benefits:

  • Supports APIs that return multiple files (e.g., document splits, image batches).
  • Backward-compatible with current single-file responses.

Additional Notes:

  • Dependencies: Requires requests-toolbelt (already in use elsewhere?).
  • Edge cases: Should invalid multipart data fall back to raw binary?

Let me know if this aligns with the project’s direction—I’m happy to submit a PR!

Why This Matters:

  • User Impact: Enables workflows like "download a ZIP and process its contents as separate files".
  • Consistency: Matches how tools like Postman/cURL handle multipart.

This keeps the change minimal while addressing a real-world use case. Would love feedback!

2. Additional context or comments

Service Overview: Object Detection & Image Cropping API

My service performs object detection on input images and returns two components simultaneously via a single multipart/form-data response:

  1. Detection Metadata

    • JSON-formatted results (detected objects, bounding boxes, confidence scores)
    • Example: 
      {
  "status": "success",
  "objects": [
    {"label": "dog", "confidence": 0.92, "bbox": [x1,y1,x2,y2]},
    {"label": "cup", "confidence": 0.87, "bbox": [x1,y1,x2,y2]}
  ]
}

  2. Cropped Image Files

    • Individual image crops for each detected object
    • Named sequentially (e.g., crop_0.jpg, crop_1.jpg)
    • MIME type: image/jpeg or image/png

Response Example

HTTP/1.1 200 OK
Content-Type: multipart/form-data; boundary=----Boundary1234

----Boundary1234
Content-Disposition: form-data; name="metadata"
Content-Type: application/json

{"status":"success","objects":[{"label":"dog",...}]}
----Boundary1234
Content-Disposition: form-data; name="crop_0"; filename="dog.jpg"
Content-Type: image/jpeg

[binary image data]
----Boundary1234
Content-Disposition: form-data; name="crop_1"; filename="cup.jpg"
Content-Type: image/jpeg

[binary image data]
----Boundary1234--

Key Characteristics

  • Atomic operation: Detection + cropping in one call
  • Preserved spatial relationships: Crops match metadata ordering
  • Scalable: Handles 50+ crops/request (tested with 10MB total payload)

Integration Challenges Addressed

  1. Clients must parse both structured data (JSON) and binary files atomically
  2. Requires proper Content-Disposition handling for filename preservation
  3. Memory-efficient streaming for large crop batches

Suggested Client Implementation

from requests_toolbelt.multipart import decoder

response = requests.post(API_URL, files={"image": open("input.jpg", "rb")})
multipart_data = decoder.MultipartDecoder.from_response(response)

results = {}
for part in multipart_data.parts:
    name = part.headers[b'Content-Disposition'].split(b'name="')[1].split(b'"')[0].decode()
    if name == "metadata":
        results[name] = json.loads(part.content.decode())
    elif name.startswith("crop_"):
        results[name] = part.content  # or save to disk

3. Can you help us with this feature?

  • I am interested in contributing to this feature.
Originally created by @signhuwei on GitHub (Apr 22, 2025). ### Self Checks - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:) - [x] Please do not modify this template :) and fill in all the required fields. ### 1. Is this request related to a challenge you're experiencing? Tell me about your story. **Description**: Currently, the `extract_files` method in `HttpRequestNode` processes HTTP responses as a single file (even for `multipart` responses). This limits scenarios where the third-party API returns multiple files (e.g., batch image crops or `multipart/form-data`). ### Proposed Changes: 1. **Detect `multipart` responses**: Check `Content-Type` for `multipart/` and parse each part using `requests_toolbelt.MultipartDecoder`. 2. **Handle each file individually**: For each part with `filename` in `Content-Disposition`, create a separate `File` object via `ToolFileManager`. 3. **Fallback to current logic**: Retain existing behavior for non-`multipart` responses (single file). ### Example Modified Code: ```python from requests_toolbelt.multipart import decoder def extract_files(self, url: str, response: Response) -> list[File]: if 'multipart/' in response.headers.get('Content-Type', ''): multipart_data = decoder.MultipartDecoder.from_response(response) for part in multipart_data.parts: if b'filename=' in part.headers.get(b'Content-Disposition', b''): # Create File for each part ... elif response.is_file: # Existing single-file logic ... ``` ### Benefits: - Supports APIs that return multiple files (e.g., document splits, image batches). - Backward-compatible with current single-file responses. ### Additional Notes: - Dependencies: Requires `requests-toolbelt` (already in use elsewhere?). - Edge cases: Should invalid `multipart` data fall back to raw binary? Let me know if this aligns with the project’s direction—I’m happy to submit a PR! --- ### Why This Matters: - **User Impact**: Enables workflows like "download a ZIP and process its contents as separate files". - **Consistency**: Matches how tools like Postman/cURL handle `multipart`. This keeps the change minimal while addressing a real-world use case. Would love feedback! ### 2. Additional context or comments **Service Overview: Object Detection & Image Cropping API** My service performs object detection on input images and returns two components simultaneously via a single `multipart/form-data` response: 1. **Detection Metadata** - JSON-formatted results (detected objects, bounding boxes, confidence scores) - Example: ```json { "status": "success", "objects": [ {"label": "dog", "confidence": 0.92, "bbox": [x1,y1,x2,y2]}, {"label": "cup", "confidence": 0.87, "bbox": [x1,y1,x2,y2]} ] } ``` 2. **Cropped Image Files** - Individual image crops for each detected object - Named sequentially (e.g., `crop_0.jpg`, `crop_1.jpg`) - MIME type: `image/jpeg` or `image/png` **Response Example** ```http HTTP/1.1 200 OK Content-Type: multipart/form-data; boundary=----Boundary1234 ----Boundary1234 Content-Disposition: form-data; name="metadata" Content-Type: application/json {"status":"success","objects":[{"label":"dog",...}]} ----Boundary1234 Content-Disposition: form-data; name="crop_0"; filename="dog.jpg" Content-Type: image/jpeg [binary image data] ----Boundary1234 Content-Disposition: form-data; name="crop_1"; filename="cup.jpg" Content-Type: image/jpeg [binary image data] ----Boundary1234-- ``` **Key Characteristics** - Atomic operation: Detection + cropping in one call - Preserved spatial relationships: Crops match metadata ordering - Scalable: Handles 50+ crops/request (tested with 10MB total payload) **Integration Challenges Addressed** 1. Clients must parse both structured data (JSON) and binary files atomically 2. Requires proper `Content-Disposition` handling for filename preservation 3. Memory-efficient streaming for large crop batches **Suggested Client Implementation** ```python from requests_toolbelt.multipart import decoder response = requests.post(API_URL, files={"image": open("input.jpg", "rb")}) multipart_data = decoder.MultipartDecoder.from_response(response) results = {} for part in multipart_data.parts: name = part.headers[b'Content-Disposition'].split(b'name="')[1].split(b'"')[0].decode() if name == "metadata": results[name] = json.loads(part.content.decode()) elif name.startswith("crop_"): results[name] = part.content # or save to disk ``` ### 3. Can you help us with this feature? - [x] I am interested in contributing to this feature.
yindo added the 💪 enhancement label 2026-02-21 19:11:11 -05:00
yindo closed this issue 2026-02-21 19:11:11 -05:00
yindo commented 2026-02-21 19:11:11 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (May 23, 2025):

Hi, @signhuwei. I'm Dosu, and I'm helping the Dify team manage their backlog. I'm marking this issue as stale.

Issue Summary

  • Enhancement proposal for HttpRequestNode.extract_files method.
  • Aim: Enable multi-file extraction from multipart HTTP responses.
  • No comments or activity on the issue since it was opened.

Next Steps

  • Is this issue still relevant to the latest version of the Dify repository? If so, please comment to keep the discussion open.
  • Otherwise, this issue will be automatically closed in 15 days.

Thank you for your understanding and contribution!

@dosubot[bot] commented on GitHub (May 23, 2025): Hi, @signhuwei. I'm [Dosu](https://dosu.dev), and I'm helping the Dify team manage their backlog. I'm marking this issue as stale. **Issue Summary** - Enhancement proposal for `HttpRequestNode.extract_files` method. - Aim: Enable multi-file extraction from `multipart` HTTP responses. - No comments or activity on the issue since it was opened. **Next Steps** - Is this issue still relevant to the latest version of the Dify repository? If so, please comment to keep the discussion open. - Otherwise, this issue will be automatically closed in 15 days. Thank you for your understanding and contribution!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 💪 enhancement
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#13231
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?