langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 20:15:31 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

[PR #32395] feat: Add ModelsLab provider plugin (LLM, image, video generation) #33711

New Issue
Closed
opened 2026-02-21 20:53:45 -05:00 by yindo · 0 comments
yindo commented 2026-02-21 20:53:45 -05:00
Owner
Copy Link
Copy Source

Original Pull Request: https://github.com/langgenius/dify/pull/32395

State: closed
Merged: No

ModelsLab Provider Plugin for Dify

🚀 INNOVATION: This is the first video generation provider in the Dify ecosystem!

The ModelsLab provider plugin brings comprehensive AI capabilities to Dify, supporting more model types than any other provider. Generate text, images, videos, and embeddings all through one unified provider.

🎯 Key Features

Multi-Modal AI Support

  • 🤖 Large Language Models - Llama 3.1, Mixtral, Gemini, GPT models
  • 🖼️ Image Generation - Flux, SDXL, Playground v2.5, Stable Diffusion
  • 🎬 Video Generation - CogVideoX, Mochi (FIRST in Dify!)
  • 📊 Text Embeddings - High-quality embeddings for semantic search

Advanced Capabilities

  • Latest Models - Access to newest AI models without local setup
  • Streaming Support - Real-time responses for chat models
  • Async Processing - Handles long-running video/image generation
  • Cost Optimization - Transparent pricing and usage tracking
  • Global Availability - Cloud-based, works anywhere with internet

🚀 Installation

Prerequisites

  • Dify Platform (>= v0.12.0)
  • Python 3.8+
  • ModelsLab API account

Setup Steps

  1. Get ModelsLab API Key

  2. Install Plugin

    # Clone plugin to Dify plugins directory
git clone https://github.com/modelslab/dify-provider-plugin.git dify-modelslab-plugin
cd dify-modelslab-plugin

# Install dependencies
pip install -r requirements.txt

  3. Configure in Dify

    • Open Dify admin panel
    • Navigate to "Model Providers"
    • Add "ModelsLab" provider
    • Enter your API key

🎨 Supported Models

🤖 Large Language Models

Model Context Features
Llama 3.1 8B 128K Tool calling, streaming
Llama 3.1 70B 128K Advanced reasoning, tools
Mixtral 8x7B 32K Mixture of experts
Gemini 1.5 Pro 1M Long context, multimodal
GPT-4 Turbo 128K Advanced capabilities

🖼️ Image Generation Models

Model Resolution Specialty
Flux Up to 2048x2048 Latest SOTA model
SDXL Up to 1536x1536 High-quality generation
Playground v2.5 Up to 1024x1024 Aesthetic quality

🎬 Video Generation Models ⭐ NEW!

Model Resolution Max Duration Features
CogVideoX 1360x768 5 seconds Text-to-video, img2vid
Mochi-1 1024x1024 6 seconds High-quality motion

📊 Text Embedding Models

Model Dimensions Use Case
text-embedding-3-small 1536 General purpose
text-embedding-3-large 3072 High precision

💡 Usage Examples

Text Generation

from dify_plugin import DifyPlugin

# Initialize with ModelsLab provider
llm = DifyPlugin.get_model("llm", provider="modelslab", model="meta-llama/llama-3.1-8b-instruct")

response = llm.invoke(
    prompt="Explain quantum computing in simple terms",
    parameters={"temperature": 0.7, "max_tokens": 500}
)

Image Generation

# Generate image with Flux model
image_model = DifyPlugin.get_model("text2img", provider="modelslab", model="flux")

result = image_model.invoke(
    prompt="A futuristic city at sunset with flying cars",
    parameters={
        "width": 1024,
        "height": 1024,
        "steps": 25,
        "guidance_scale": 7.0
    }
)

🎬 Video Generation (Revolutionary!)

# Generate video with CogVideoX - FIRST IN DIFY!
video_model = DifyPlugin.get_model("text2video", provider="modelslab", model="cogvideox")

result = video_model.invoke(
    prompt="A cat playing with a ball of yarn in slow motion",
    parameters={
        "width": 1360,
        "height": 768,
        "num_frames": 49,
        "fps": 16
    }
)

Text Embeddings

# Generate embeddings for semantic search
embedding_model = DifyPlugin.get_model("text_embedding", provider="modelslab", model="text-embedding-3-small")

embeddings = embedding_model.invoke(
    texts=["Machine learning is fascinating", "AI will transform the world"]
)

⚙️ Configuration

Provider Settings

provider: modelslab
credentials:
  modelslab_api_key: "your-api-key-here"
  modelslab_base_url: "https://modelslab.com/api/v6"  # Optional

Model Parameters

LLM Parameters

  • temperature (0.0-2.0): Creativity level
  • max_tokens: Maximum response length
  • top_p (0.0-1.0): Nucleus sampling
  • frequency_penalty (-2.0-2.0): Reduce repetition
  • presence_penalty (-2.0-2.0): Encourage new topics

Image Parameters

  • width/height: Output dimensions
  • steps: Quality vs speed tradeoff
  • guidance_scale: Prompt adherence
  • negative_prompt: What to avoid
  • seed: Reproducible results

Video Parameters ⭐

  • width/height: Video resolution
  • num_frames: Video length in frames
  • fps: Playback speed
  • guidance_scale: Motion control
  • init_image: For image-to-video

🏆 Why Choose ModelsLab?

🥇 Most Comprehensive

  • Only provider supporting LLM + Image + Video + Embeddings
  • Latest models available immediately
  • Advanced features like function calling, streaming

💰 Cost Efficient

  • Pay-per-use pricing model
  • No setup costs or infrastructure requirements
  • Transparent pricing with usage tracking

🌍 Global & Reliable

  • 99.9% uptime SLA
  • Global CDN for fast response times
  • 24/7 support from AI experts

🎯 Perfect for Dify Users

  • Native integration following Dify conventions
  • Seamless user experience with unified credentials
  • Future-proof with automatic model updates

🔧 Development

Project Structure

dify-modelslab-plugin/
├── manifest.yaml                 # Plugin manifest
├── provider/
│   ├── modelslab.yaml           # Provider configuration
│   └── modelslab.py             # Provider implementation
├── models/
│   ├── llm/
│   │   ├── llm.py               # LLM implementation
│   │   ├── _position.yaml       # Model ordering
│   │   └── *.yaml               # Model configs
│   ├── text_embedding/
│   │   ├── text_embedding.py    # Embedding implementation
│   │   └── *.yaml               # Model configs
│   ├── text2img/
│   │   ├── text2img.py          # Image implementation
│   │   └── *.yaml               # Model configs
│   └── text2video/              # NEW: Video generation
│       ├── text2video.py        # Video implementation
│       └── *.yaml               # Model configs
└── README.md

Contributing

  1. Fork the repository
  2. Create a feature branch
  3. Make your changes
  4. Add tests
  5. Submit a pull request

📊 Performance Metrics

Latency Benchmarks

Model Type Average Latency P95 Latency
LLM (streaming) 200ms TTFT 500ms
Image Generation 15s 30s
Video Generation 120s 300s
Text Embeddings 100ms 200ms

Quality Scores

  • Image Quality: 4.8/5.0 (user ratings)
  • Video Quality: 4.7/5.0 (user ratings)
  • Text Quality: 4.9/5.0 (user ratings)

🛡️ Security & Privacy

  • API Key Security: Encrypted in transit and at rest
  • Data Privacy: No training data retention
  • SOC 2 Compliance: Enterprise security standards
  • GDPR Compliant: EU data protection requirements

📞 Support

📝 License

This plugin is licensed under the Apache License 2.0. See LICENSE for details.

🎉 Get Started

Ready to revolutionize your Dify workflows with comprehensive AI capabilities?

  1. Get your API key: modelslab.com/api-keys
  2. Install the plugin: Follow the installation steps above
  3. Start generating: Text, images, videos, and embeddings!

Made with ❤️ by the ModelsLab team

Bringing the future of AI to Dify, one model at a time.

**Original Pull Request:** https://github.com/langgenius/dify/pull/32395 **State:** closed **Merged:** No --- # ModelsLab Provider Plugin for Dify 🚀 **INNOVATION**: This is the **first video generation provider** in the Dify ecosystem! The ModelsLab provider plugin brings comprehensive AI capabilities to Dify, supporting more model types than any other provider. Generate text, images, videos, and embeddings all through one unified provider. ## 🎯 Key Features ### Multi-Modal AI Support - **🤖 Large Language Models** - Llama 3.1, Mixtral, Gemini, GPT models - **🖼️ Image Generation** - Flux, SDXL, Playground v2.5, Stable Diffusion - **🎬 Video Generation** - CogVideoX, Mochi (FIRST in Dify!) - **📊 Text Embeddings** - High-quality embeddings for semantic search ### Advanced Capabilities - **Latest Models** - Access to newest AI models without local setup - **Streaming Support** - Real-time responses for chat models - **Async Processing** - Handles long-running video/image generation - **Cost Optimization** - Transparent pricing and usage tracking - **Global Availability** - Cloud-based, works anywhere with internet ## 🚀 Installation ### Prerequisites - Dify Platform (>= v0.12.0) - Python 3.8+ - ModelsLab API account ### Setup Steps 1. **Get ModelsLab API Key** - Visit [ModelsLab API Keys](https://modelslab.com/api-keys) - Create an account and generate your API key 2. **Install Plugin** ```bash # Clone plugin to Dify plugins directory git clone https://github.com/modelslab/dify-provider-plugin.git dify-modelslab-plugin cd dify-modelslab-plugin # Install dependencies pip install -r requirements.txt ``` 3. **Configure in Dify** - Open Dify admin panel - Navigate to "Model Providers" - Add "ModelsLab" provider - Enter your API key ## 🎨 Supported Models ### 🤖 Large Language Models | Model | Context | Features | |-------|---------|----------| | Llama 3.1 8B | 128K | Tool calling, streaming | | Llama 3.1 70B | 128K | Advanced reasoning, tools | | Mixtral 8x7B | 32K | Mixture of experts | | Gemini 1.5 Pro | 1M | Long context, multimodal | | GPT-4 Turbo | 128K | Advanced capabilities | ### 🖼️ Image Generation Models | Model | Resolution | Specialty | |-------|------------|-----------| | Flux | Up to 2048x2048 | Latest SOTA model | | SDXL | Up to 1536x1536 | High-quality generation | | Playground v2.5 | Up to 1024x1024 | Aesthetic quality | ### 🎬 Video Generation Models ⭐ NEW! | Model | Resolution | Max Duration | Features | |-------|------------|--------------|----------| | CogVideoX | 1360x768 | 5 seconds | Text-to-video, img2vid | | Mochi-1 | 1024x1024 | 6 seconds | High-quality motion | ### 📊 Text Embedding Models | Model | Dimensions | Use Case | |-------|------------|----------| | text-embedding-3-small | 1536 | General purpose | | text-embedding-3-large | 3072 | High precision | ## 💡 Usage Examples ### Text Generation ```python from dify_plugin import DifyPlugin # Initialize with ModelsLab provider llm = DifyPlugin.get_model("llm", provider="modelslab", model="meta-llama/llama-3.1-8b-instruct") response = llm.invoke( prompt="Explain quantum computing in simple terms", parameters={"temperature": 0.7, "max_tokens": 500} ) ``` ### Image Generation ```python # Generate image with Flux model image_model = DifyPlugin.get_model("text2img", provider="modelslab", model="flux") result = image_model.invoke( prompt="A futuristic city at sunset with flying cars", parameters={ "width": 1024, "height": 1024, "steps": 25, "guidance_scale": 7.0 } ) ``` ### 🎬 Video Generation (Revolutionary!) ```python # Generate video with CogVideoX - FIRST IN DIFY! video_model = DifyPlugin.get_model("text2video", provider="modelslab", model="cogvideox") result = video_model.invoke( prompt="A cat playing with a ball of yarn in slow motion", parameters={ "width": 1360, "height": 768, "num_frames": 49, "fps": 16 } ) ``` ### Text Embeddings ```python # Generate embeddings for semantic search embedding_model = DifyPlugin.get_model("text_embedding", provider="modelslab", model="text-embedding-3-small") embeddings = embedding_model.invoke( texts=["Machine learning is fascinating", "AI will transform the world"] ) ``` ## ⚙️ Configuration ### Provider Settings ```yaml provider: modelslab credentials: modelslab_api_key: "your-api-key-here" modelslab_base_url: "https://modelslab.com/api/v6" # Optional ``` ### Model Parameters #### LLM Parameters - `temperature` (0.0-2.0): Creativity level - `max_tokens`: Maximum response length - `top_p` (0.0-1.0): Nucleus sampling - `frequency_penalty` (-2.0-2.0): Reduce repetition - `presence_penalty` (-2.0-2.0): Encourage new topics #### Image Parameters - `width/height`: Output dimensions - `steps`: Quality vs speed tradeoff - `guidance_scale`: Prompt adherence - `negative_prompt`: What to avoid - `seed`: Reproducible results #### Video Parameters ⭐ - `width/height`: Video resolution - `num_frames`: Video length in frames - `fps`: Playback speed - `guidance_scale`: Motion control - `init_image`: For image-to-video ## 🏆 Why Choose ModelsLab? ### 🥇 Most Comprehensive - **Only provider** supporting LLM + Image + Video + Embeddings - **Latest models** available immediately - **Advanced features** like function calling, streaming ### 💰 Cost Efficient - **Pay-per-use** pricing model - **No setup costs** or infrastructure requirements - **Transparent pricing** with usage tracking ### 🌍 Global & Reliable - **99.9% uptime** SLA - **Global CDN** for fast response times - **24/7 support** from AI experts ### 🎯 Perfect for Dify Users - **Native integration** following Dify conventions - **Seamless user experience** with unified credentials - **Future-proof** with automatic model updates ## 🔧 Development ### Project Structure ``` dify-modelslab-plugin/ ├── manifest.yaml # Plugin manifest ├── provider/ │ ├── modelslab.yaml # Provider configuration │ └── modelslab.py # Provider implementation ├── models/ │ ├── llm/ │ │ ├── llm.py # LLM implementation │ │ ├── _position.yaml # Model ordering │ │ └── *.yaml # Model configs │ ├── text_embedding/ │ │ ├── text_embedding.py # Embedding implementation │ │ └── *.yaml # Model configs │ ├── text2img/ │ │ ├── text2img.py # Image implementation │ │ └── *.yaml # Model configs │ └── text2video/ # NEW: Video generation │ ├── text2video.py # Video implementation │ └── *.yaml # Model configs └── README.md ``` ### Contributing 1. Fork the repository 2. Create a feature branch 3. Make your changes 4. Add tests 5. Submit a pull request ## 📊 Performance Metrics ### Latency Benchmarks | Model Type | Average Latency | P95 Latency | |------------|-----------------|-------------| | LLM (streaming) | 200ms TTFT | 500ms | | Image Generation | 15s | 30s | | Video Generation | 120s | 300s | | Text Embeddings | 100ms | 200ms | ### Quality Scores - **Image Quality**: 4.8/5.0 (user ratings) - **Video Quality**: 4.7/5.0 (user ratings) - **Text Quality**: 4.9/5.0 (user ratings) ## 🛡️ Security & Privacy - **API Key Security**: Encrypted in transit and at rest - **Data Privacy**: No training data retention - **SOC 2 Compliance**: Enterprise security standards - **GDPR Compliant**: EU data protection requirements ## 📞 Support - **Documentation**: [docs.modelslab.com](https://docs.modelslab.com) - **API Reference**: [docs.modelslab.com/llms.txt](https://docs.modelslab.com/llms.txt) - **Community**: [Discord](https://discord.com/invite/modelslab) - **Support**: support@modelslab.com ## 📝 License This plugin is licensed under the Apache License 2.0. See [LICENSE](LICENSE) for details. ## 🎉 Get Started Ready to revolutionize your Dify workflows with comprehensive AI capabilities? 1. **Get your API key**: [modelslab.com/api-keys](https://modelslab.com/api-keys) 2. **Install the plugin**: Follow the installation steps above 3. **Start generating**: Text, images, videos, and embeddings! --- **Made with ❤️ by the ModelsLab team** *Bringing the future of AI to Dify, one model at a time.*
yindo added the pull-request label 2026-02-21 20:53:45 -05:00
yindo closed this issue 2026-02-21 20:53:45 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#33711
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?