langgenius/dify-official-plugins
1
0
Fork 0
mirror of https://github.com/langgenius/dify-official-plugins.git synced 2026-07-22 01:55:27 -04:00
Code Issues 54 Packages Projects Releases Wiki Activity

[PR #2574] [MERGED] feat: Add vision support for text embedding and reranker model #2580

New Issue
Closed
opened 2026-02-16 11:17:22 -05:00 by yindo · 0 comments
yindo commented 2026-02-16 11:17:22 -05:00
Owner
Copy Link
Copy Source

📋 Pull Request Information

Original PR: https://github.com/langgenius/dify-official-plugins/pull/2574
Author: @PaimonLumine
Created: 2/11/2026
Status: ✅ Merged
Merged: 2/11/2026
Merged by: @crazywoola

Base: main ← Head: main

📝 Commits (5)

  • 3e1e501 feat(openai_api_compatible): add multimodal vision support to rerank
  • 0528c59 Merge branch 'langgenius:main' into main
  • 61b3622 fix(rerank): handle None features list and simplify text-only mode
  • 7509627 fix(rerank): correct top_n=0 handling and add SSRF protection
  • a843da2 Merge branch 'main' into main

📊 Changes

5 files changed (+979 additions, -15 deletions)

View changed files

📝 .gitignore (+1 -0)
📝 models/openai_api_compatible/manifest.yaml (+1 -1)
📝 models/openai_api_compatible/models/rerank/rerank.py (+423 -3)
📝 models/openai_api_compatible/models/text_embedding/text_embedding.py (+516 -11)
📝 models/openai_api_compatible/provider/openai_api_compatible.yaml (+38 -0)

📄 Description

Description

This PR enhances the openai_api_compatible plugin to add multimodal support for Qwen3-VL Embedding and Qwen3-VL Reranker models via custom base URL configuration.

Key Changes

  1. Text Embedding Model (text_embedding.py)

    • Added multimodal input support for vision-enabled models (text + image)
    • Implemented automatic text chunking to handle context length limits (configurable via max_tokens)
    • Added vision_support credential option to enable multimodal capabilities
    • Fixed EmbeddingUsage Pydantic validation errors by adding all required fields
    • Supports various image formats: URLs, base64, local file paths, markdown image syntax

  2. Rerank Model (rerank.py)

    • Added _invoke_multimodal method following vLLM/Qwen3-VL-Reranker API specification
    • Implemented ScoreMultiModalParam format for documents: 
      {"content": [
  {"type": "text", "text": "..."},
  {"type": "image_url", "image_url": {"url": "..."}}
]}
    • Query is limited to text-only (as per Qwen3-VL-Reranker single-tower architecture)
    • Documents support both text and image multimodal inputs
    • Added vision_support credential option

  3. Both Models

    • Added endpoint_model_name credential to support custom model naming
    • Enhanced error logging for debugging API requests
    • Maintained backward compatibility with text-only mode

Supported Use Cases

  • Text Query + Text Documents: Standard reranking with text inputs
  • Text Query + Multimodal Documents: Query text against documents containing both text and images
  • Multimodal Embeddings: Generate embeddings for text+image combinations

Configuration

Users can enable vision support by setting:

  • vision_support: "support" (enables multimodal capabilities)
  • endpoint_url: Custom API endpoint (e.g., http://baseurl.com)
  • endpoint_model_name: Model identifier for the endpoint

Related Issues or Context

This enhancement allows Dify users to leverage Qwen3-VL models deployed via vLLM or compatible inference engines for:

  • Multimodal retrieval (text-to-image, image-to-text)
  • Cross-modal reranking
  • Vision-enabled RAG workflows

This PR contains Changes to Non-Plugin

  • Documentation
  • Other: Enhanced existing plugin capabilities

This PR contains Changes to Non-LLM Models Plugin

  • I have Run Comprehensive Tests Relevant to My Changes

This PR contains Changes to LLM Models Plugin

  • My Changes Affect Message Flow Handling (System Messages and User→Assistant Turn-Taking)
  • My Changes Affect Tool Interaction Flow (Multi-Round Usage and Output Handling, for both Agent App and Agent Node)
  • My Changes Affect Multimodal Input Handling (Images, PDFs, Audio, Video, etc.)
    • Added support for image inputs in embedding and rerank models
    • Supports URL, base64, and file path formats
  • My Changes Affect Multimodal Output Generation (Images, Audio, Video, etc.)
  • My Changes Affect Structured Output Format (JSON, XML, etc.)
  • My Changes Affect Token Consumption Metrics
  • My Changes Affect Other LLM Functionalities (Reasoning Process, Grounding, Prompt Caching, etc.)
  • Other Changes (Add New Models, Fix Model Parameters etc.)
    • Fixed EmbeddingUsage validation errors
    • Added automatic text chunking for long inputs
    • Enhanced multimodal reranker support

Version Control (Any Changes to the Plugin Will Require Bumping the Version)

  • I have Bumped Up the Version in Manifest.yaml (Top-Level Version Field, Not in Meta Section)
    • Suggested version bump: 0.0.35 (minor feature addition)

Dify Plugin SDK Version

  • I have Ensured dify_plugin>=0.3.0,<0.6.0 is in requirements.txt

Environment Verification (If Any Code Changes)

Local Deployment Environment

  • Dify Version is: 1.12.1, I have Tested My Changes on Local Deployment Dify with a Clean Environment That Matches the Production Configuration.
    • Tested with Qwen3-VL-Embedding-2B via vLLM
    • Tested with Qwen3-VL-Reranker-2B via vLLM
    • Verified multimodal inputs (text + images)
    • Verified automatic text chunking for long documents
    • Verified backward compatibility with text-only mode

SaaS Environment

  • I have Tested My Changes on cloud.dify.ai with a Clean Environment That Matches the Production Configuration

This is not tested because my self-hosted models are private.

Screenshots/Demonstrations

Multimodal Embedding Test

# Example: Text + Image embedding
inputs = [
    '{"text": "A woman playing with her dog", "image": "https://example.com/image.jpg"}'
]

Multimodal Rerank Test

# Example: Text query with multimodal documents
query = "A woman playing with her dog on a beach"
docs = [
    "A woman shares a joyful moment with her golden retriever...",
    '{"image": "https://example.com/beach.jpg"}',
    '{"text": "A woman with her dog", "image": "https://example.com/dog.jpg"}'
]

Testing Checklist

  • Text embedding (backward compatibility)
  • Multimodal embedding (text + image)
  • Text-only reranking
  • Multimodal reranking (text query + multimodal docs)
  • Automatic text chunking for long inputs
  • Error handling for invalid image formats
  • Credential validation with custom endpoints

Known Issues

  1. Qwen3-VL-Reranker Image Query Not Supported in Retrieval Test:
    • Dify's built-in Retrieval Test feature sends queries as pure text strings
    • Due to Qwen3-VL-Reranker's single-tower architecture, image queries are not supported (query must be text-only)
    • Workaround: Use the reranker in actual chat workflows where documents can contain images while queries remain text-only
    • For pure image-to-image or image-to-text retrieval, consider using the multimodal embedding model instead

Notes

  1. Qwen3-VL-Reranker Limitation: The model uses a single-tower architecture, so queries must be text-only. Documents can be multimodal.

  2. Image Processing: The plugin automatically handles:

    • HTTP/HTTPS URLs
    • Base64 encoded images
    • Local file paths (converted to file://)
    • Markdown image syntax

  3. Text Chunking: For embedding models with token limits, the plugin automatically:

    • Calculates approximate token count
    • Chunks long text at sentence boundaries
    • Averages embeddings across chunks

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/langgenius/dify-official-plugins/pull/2574 **Author:** [@PaimonLumine](https://github.com/PaimonLumine) **Created:** 2/11/2026 **Status:** ✅ Merged **Merged:** 2/11/2026 **Merged by:** [@crazywoola](https://github.com/crazywoola) **Base:** `main` ← **Head:** `main` --- ### 📝 Commits (5) - [`3e1e501`](https://github.com/langgenius/dify-official-plugins/commit/3e1e501c5eee487c8b10a732287cb90cd3b2c47a) feat(openai_api_compatible): add multimodal vision support to rerank - [`0528c59`](https://github.com/langgenius/dify-official-plugins/commit/0528c59a7e377aafd41d9109246244ed3a363461) Merge branch 'langgenius:main' into main - [`61b3622`](https://github.com/langgenius/dify-official-plugins/commit/61b36223968f2b9145f5eaaca9e8d2a7f608b448) fix(rerank): handle None features list and simplify text-only mode - [`7509627`](https://github.com/langgenius/dify-official-plugins/commit/7509627f301d34e46e59476eae70b1f2b72dadd9) fix(rerank): correct top_n=0 handling and add SSRF protection - [`a843da2`](https://github.com/langgenius/dify-official-plugins/commit/a843da21b1bcd6ca4735bd48321cb1af7573e9a4) Merge branch 'main' into main ### 📊 Changes **5 files changed** (+979 additions, -15 deletions) <details> <summary>View changed files</summary> 📝 `.gitignore` (+1 -0) 📝 `models/openai_api_compatible/manifest.yaml` (+1 -1) 📝 `models/openai_api_compatible/models/rerank/rerank.py` (+423 -3) 📝 `models/openai_api_compatible/models/text_embedding/text_embedding.py` (+516 -11) 📝 `models/openai_api_compatible/provider/openai_api_compatible.yaml` (+38 -0) </details> ### 📄 Description ## Description This PR enhances the `openai_api_compatible` plugin to add multimodal support for Qwen3-VL Embedding and Qwen3-VL Reranker models via custom base URL configuration. ### Key Changes 1. **Text Embedding Model (`text_embedding.py`)** - Added multimodal input support for vision-enabled models (text + image) - Implemented automatic text chunking to handle context length limits (configurable via `max_tokens`) - Added `vision_support` credential option to enable multimodal capabilities - Fixed `EmbeddingUsage` Pydantic validation errors by adding all required fields - Supports various image formats: URLs, base64, local file paths, markdown image syntax 2. **Rerank Model (`rerank.py`)** - Added `_invoke_multimodal` method following vLLM/Qwen3-VL-Reranker API specification - Implemented `ScoreMultiModalParam` format for documents: ```json {"content": [ {"type": "text", "text": "..."}, {"type": "image_url", "image_url": {"url": "..."}} ]} ``` - Query is limited to text-only (as per Qwen3-VL-Reranker single-tower architecture) - Documents support both text and image multimodal inputs - Added `vision_support` credential option 3. **Both Models** - Added `endpoint_model_name` credential to support custom model naming - Enhanced error logging for debugging API requests - Maintained backward compatibility with text-only mode ### Supported Use Cases - **Text Query + Text Documents**: Standard reranking with text inputs - **Text Query + Multimodal Documents**: Query text against documents containing both text and images - **Multimodal Embeddings**: Generate embeddings for text+image combinations ### Configuration Users can enable vision support by setting: - `vision_support`: "support" (enables multimodal capabilities) - `endpoint_url`: Custom API endpoint (e.g., `http://baseurl.com`) - `endpoint_model_name`: Model identifier for the endpoint ## Related Issues or Context This enhancement allows Dify users to leverage Qwen3-VL models deployed via vLLM or compatible inference engines for: - Multimodal retrieval (text-to-image, image-to-text) - Cross-modal reranking - Vision-enabled RAG workflows ## This PR contains Changes to *Non-Plugin* - [ ] Documentation - [x] Other: Enhanced existing plugin capabilities ## This PR contains Changes to *Non-LLM Models Plugin* - [x] I have Run Comprehensive Tests Relevant to My Changes ## This PR contains Changes to *LLM Models Plugin* - [ ] My Changes Affect Message Flow Handling (System Messages and User→Assistant Turn-Taking) - [ ] My Changes Affect Tool Interaction Flow (Multi-Round Usage and Output Handling, for both Agent App and Agent Node) - [x] My Changes Affect Multimodal Input Handling (Images, PDFs, Audio, Video, etc.) - Added support for image inputs in embedding and rerank models - Supports URL, base64, and file path formats - [ ] My Changes Affect Multimodal Output Generation (Images, Audio, Video, etc.) - [ ] My Changes Affect Structured Output Format (JSON, XML, etc.) - [ ] My Changes Affect Token Consumption Metrics - [ ] My Changes Affect Other LLM Functionalities (Reasoning Process, Grounding, Prompt Caching, etc.) - [x] Other Changes (Add New Models, Fix Model Parameters etc.) - Fixed `EmbeddingUsage` validation errors - Added automatic text chunking for long inputs - Enhanced multimodal reranker support ## Version Control (Any Changes to the Plugin Will Require Bumping the Version) - [x] I have Bumped Up the Version in Manifest.yaml (Top-Level `Version` Field, Not in Meta Section) - Suggested version bump: **0.0.35** (minor feature addition) ## Dify Plugin SDK Version - [x] I have Ensured `dify_plugin>=0.3.0,<0.6.0` is in requirements.txt ## Environment Verification (If Any Code Changes) ### Local Deployment Environment - [x] Dify Version is: 1.12.1, I have Tested My Changes on Local Deployment Dify with a Clean Environment That Matches the Production Configuration. - Tested with Qwen3-VL-Embedding-2B via vLLM - Tested with Qwen3-VL-Reranker-2B via vLLM - Verified multimodal inputs (text + images) - Verified automatic text chunking for long documents - Verified backward compatibility with text-only mode ### SaaS Environment - [ ] I have Tested My Changes on cloud.dify.ai with a Clean Environment That Matches the Production Configuration This is not tested because my self-hosted models are private. ## Screenshots/Demonstrations ### Multimodal Embedding Test ```python # Example: Text + Image embedding inputs = [ '{"text": "A woman playing with her dog", "image": "https://example.com/image.jpg"}' ] ``` ### Multimodal Rerank Test ```python # Example: Text query with multimodal documents query = "A woman playing with her dog on a beach" docs = [ "A woman shares a joyful moment with her golden retriever...", '{"image": "https://example.com/beach.jpg"}', '{"text": "A woman with her dog", "image": "https://example.com/dog.jpg"}' ] ``` ## Testing Checklist - [x] Text embedding (backward compatibility) - [x] Multimodal embedding (text + image) - [x] Text-only reranking - [x] Multimodal reranking (text query + multimodal docs) - [x] Automatic text chunking for long inputs - [x] Error handling for invalid image formats - [x] Credential validation with custom endpoints ## Known Issues 1. **Qwen3-VL-Reranker Image Query Not Supported in Retrieval Test**: - Dify's built-in Retrieval Test feature sends queries as pure text strings - Due to Qwen3-VL-Reranker's single-tower architecture, image queries are not supported (query must be text-only) - **Workaround**: Use the reranker in actual chat workflows where documents can contain images while queries remain text-only - For pure image-to-image or image-to-text retrieval, consider using the multimodal embedding model instead ## Notes 1. **Qwen3-VL-Reranker Limitation**: The model uses a single-tower architecture, so queries must be text-only. Documents can be multimodal. 2. **Image Processing**: The plugin automatically handles: - HTTP/HTTPS URLs - Base64 encoded images - Local file paths (converted to file://) - Markdown image syntax 3. **Text Chunking**: For embedding models with token limits, the plugin automatically: - Calculates approximate token count - Chunks long text at sentence boundaries - Averages embeddings across chunks --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
yindo added the pull-request label 2026-02-16 11:17:22 -05:00
yindo closed this issue 2026-02-16 11:17:22 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
bug
bug
dependencies
documentation
enhancement
good first issue
invalid
pull-request
Mirrored from GitHub Pull Request
 question
stale
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify-official-plugins#2580
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?