langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 12:05:30 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Generate image error #4913

New Issue
Closed
opened 2026-02-21 18:08:35 -05:00 by yindo · 1 comment
yindo commented 2026-02-21 18:08:35 -05:00
Owner
Copy Link
Copy Source

Originally created by @guleng on GitHub (Aug 6, 2024).

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

0.6.16

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

I created a custom tool that can connect to the Litellm gateway to create a model of text and graphics. When the graphics are generated, they are returned in b64_json format and cannot be displayed directly if they are not binary. The second issue is that the returned string is too large and restricted.

Because all models are locally deployed and load balanced through Litellm, when adding the text graph model to dify, Stable Diffusion is used. It does not have a place to write the Litellm token or specify the model name, so I can only create a custom tool myself. I have upgraded to the latest version and there is no other way to deal with this problem. Is there any way to add the Litellm text graph model?

✔️ Expected Behavior

openapi: 3.1.0
info:
  title: Dify Custom Image Generation API
  version: 1.0.0
servers:
  - url: https://litellm.fastpod.ai/v1

components:
  schemas:
    ImageGenerationRequest:
      type: object
      properties:
        model:
          type: string
          const: dall-e-3
        prompt:
          type: string
          description: A text description of the desired image.
          maxLength: 2048
        size:
          type: string
          description: The size of the generated image, default is 1024x1024.
          enum: ["1024x1024", "720x1280", "1280x720"]
      required:
        - prompt

    ImageGenerationResponse:
      type: object
      properties:
        created:
          type: integer
          format: int64
          description: The timestamp of when the image was created.
        data:
          type: array
          items:
            type: object
            properties:
              b64_json:
                type: string
                format: byte
                description: Base64 encoded image data.
                maxLength: 100000000  # 限制 Base64 编码字符串的最大长度
              detail:
                type: string
                description: Additional details about the image generation process.

  securitySchemes:
    bearerAuth:
      type: http
      scheme: bearer
      bearerFormat: JWT

paths:
  /images/generations:
    post:
      summary: Generate an image from a text prompt
      description: Creates an image from a given text prompt using the specified model.
      operationId: generateImage
      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/ImageGenerationRequest'
      responses:
        '200':
          description: Successful response
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ImageGenerationResponse'
        '400':
          description: Bad request
        '401':
          description: Unauthorized
      security:
        - bearerAuth: []

security:
  - bearerAuth: []

❌ Actual Behavior

image

Thank you.

Originally created by @guleng on GitHub (Aug 6, 2024). ### Self Checks - [X] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [X] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [X] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [X] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:） - [X] Please do not modify this template :) and fill in all the required fields. ### Dify version 0.6.16 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce I created a custom tool that can connect to the Litellm gateway to create a model of text and graphics. When the graphics are generated, they are returned in b64_json format and cannot be displayed directly if they are not binary. The second issue is that the returned string is too large and restricted. Because all models are locally deployed and load balanced through Litellm, when adding the text graph model to dify, Stable Diffusion is used. It does not have a place to write the Litellm token or specify the model name, so I can only create a custom tool myself. I have upgraded to the latest version and there is no other way to deal with this problem. Is there any way to add the Litellm text graph model? ### ✔️ Expected Behavior ``` openapi: 3.1.0 info: title: Dify Custom Image Generation API version: 1.0.0 servers: - url: https://litellm.fastpod.ai/v1 components: schemas: ImageGenerationRequest: type: object properties: model: type: string const: dall-e-3 prompt: type: string description: A text description of the desired image. maxLength: 2048 size: type: string description: The size of the generated image, default is 1024x1024. enum: ["1024x1024", "720x1280", "1280x720"] required: - prompt ImageGenerationResponse: type: object properties: created: type: integer format: int64 description: The timestamp of when the image was created. data: type: array items: type: object properties: b64_json: type: string format: byte description: Base64 encoded image data. maxLength: 100000000 # 限制 Base64 编码字符串的最大长度 detail: type: string description: Additional details about the image generation process. securitySchemes: bearerAuth: type: http scheme: bearer bearerFormat: JWT paths: /images/generations: post: summary: Generate an image from a text prompt description: Creates an image from a given text prompt using the specified model. operationId: generateImage requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/ImageGenerationRequest' responses: '200': description: Successful response content: application/json: schema: $ref: '#/components/schemas/ImageGenerationResponse' '400': description: Bad request '401': description: Unauthorized security: - bearerAuth: [] security: - bearerAuth: [] ``` ### ❌ Actual Behavior ![image](https://github.com/user-attachments/assets/99755517-9eb8-4ee0-b663-b204f77eb3a1) Thank you.
yindo added the 🐞 bug label 2026-02-21 18:08:35 -05:00
yindo closed this issue 2026-02-21 18:08:35 -05:00
yindo commented 2026-02-21 18:08:35 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Aug 6, 2024):

It looks like you're encountering two main issues with your custom tool for image generation via the Litellm gateway in Dify:

  1. Handling b64_json formatted data: The generated images are returned in b64_json format, which cannot be displayed directly if they are not binary.
  2. Payload size limitations: The returned string is too large and restricted, causing issues with the API request.

Handling b64_json formatted data

To display b64_json formatted data as images, you need to convert the Base64 encoded string to an image format that can be rendered in your application. Here is a general approach to handle this:

  1. Decode the Base64 string: Convert the Base64 string to binary data.
  2. Create an image element: Use the binary data to create an image element in your application.

Here is an example in JavaScript:

function displayBase64Image(b64_json) {
  const img = new Image();
  img.src = `data:image/png;base64,${b64_json}`;
  document.body.appendChild(img); // Append the image to the body or any other container
}

Adjusting Payload Size Limitations

Dify has configurable settings for the maximum size of API request payloads. You can adjust these settings in the .env file to accommodate larger payloads. The relevant settings are:

  • HTTP_REQUEST_NODE_MAX_BINARY_SIZE: Maximum binary size for HTTP requests (default is 10MB).
  • HTTP_REQUEST_NODE_MAX_TEXT_SIZE: Maximum text size for HTTP requests (default is 1MB).

To adjust these settings, modify the .env file in your Dify setup:

HTTP_REQUEST_NODE_MAX_BINARY_SIZE=20971520  # 20MB
HTTP_REQUEST_NODE_MAX_TEXT_SIZE=5242880     # 5MB

Integrating Litellm Text Graph Model

To integrate the Litellm text graph model and specify the token and model name, you can configure custom tokens and model names in Dify. Here is an example configuration:

addApiKey: 'Add your API key',
invalidApiKey: 'Invalid API key',
encrypted: {
  front: 'Your API key will be encrypted and stored using',
  back: 'technology.',
},
modelProvider: {
  notConfigured: 'The system model has not been fully configured yet, and some functions may be unavailable.',
  systemModelSettings: 'System Model Settings',
  selectModel: 'Select your model',
  setupModelFirst: 'Please set up your model first',
  systemReasoningModel: {
    key: 'System Reasoning Model',
    tip: 'Set the default inference model to use for app creation, as well as features such as generating dialogue names and suggesting the next question will also use the default inference model.',
  },
  embeddingModel: {
    key: 'Embedding Model',
    tip: 'Set the default model for document embedding processing in Knowledge, both retrieval and import of Knowledge use this Embedding model for vectorization processing.',
    required: 'The embedding model is required',
  },
  speechToTextModel: {
    key: 'Speech-to-Text Model',
    tip: 'Set the default model for text input by speech in the conversation.',
  },
  ttsModel: {
    key: 'Text-to-Speech Model',
  },
},

This configuration allows you to add and configure custom models and API keys, including those for external APIs like Litellm [1].

For more detailed guidance on integrating third-party APIs, you can refer to the Tool Integration Guide [2].

To continue talking to Dosu, mention @dosu.

@dosubot[bot] commented on GitHub (Aug 6, 2024): <!-- Answer --> It looks like you're encountering two main issues with your custom tool for image generation via the Litellm gateway in Dify: 1. **Handling `b64_json` formatted data**: The generated images are returned in `b64_json` format, which cannot be displayed directly if they are not binary. 2. **Payload size limitations**: The returned string is too large and restricted, causing issues with the API request. ### Handling `b64_json` formatted data To display `b64_json` formatted data as images, you need to convert the Base64 encoded string to an image format that can be rendered in your application. Here is a general approach to handle this: 1. **Decode the Base64 string**: Convert the Base64 string to binary data. 2. **Create an image element**: Use the binary data to create an image element in your application. Here is an example in JavaScript: ```javascript function displayBase64Image(b64_json) { const img = new Image(); img.src = `data:image/png;base64,${b64_json}`; document.body.appendChild(img); // Append the image to the body or any other container } ``` ### Adjusting Payload Size Limitations Dify has configurable settings for the maximum size of API request payloads. You can adjust these settings in the `.env` file to accommodate larger payloads. The relevant settings are: - `HTTP_REQUEST_NODE_MAX_BINARY_SIZE`: Maximum binary size for HTTP requests (default is 10MB). - `HTTP_REQUEST_NODE_MAX_TEXT_SIZE`: Maximum text size for HTTP requests (default is 1MB). To adjust these settings, modify the `.env` file in your Dify setup: ```env HTTP_REQUEST_NODE_MAX_BINARY_SIZE=20971520 # 20MB HTTP_REQUEST_NODE_MAX_TEXT_SIZE=5242880 # 5MB ``` ### Integrating Litellm Text Graph Model To integrate the Litellm text graph model and specify the token and model name, you can configure custom tokens and model names in Dify. Here is an example configuration: ```ts addApiKey: 'Add your API key', invalidApiKey: 'Invalid API key', encrypted: { front: 'Your API key will be encrypted and stored using', back: 'technology.', }, modelProvider: { notConfigured: 'The system model has not been fully configured yet, and some functions may be unavailable.', systemModelSettings: 'System Model Settings', selectModel: 'Select your model', setupModelFirst: 'Please set up your model first', systemReasoningModel: { key: 'System Reasoning Model', tip: 'Set the default inference model to use for app creation, as well as features such as generating dialogue names and suggesting the next question will also use the default inference model.', }, embeddingModel: { key: 'Embedding Model', tip: 'Set the default model for document embedding processing in Knowledge, both retrieval and import of Knowledge use this Embedding model for vectorization processing.', required: 'The embedding model is required', }, speechToTextModel: { key: 'Speech-to-Text Model', tip: 'Set the default model for text input by speech in the conversation.', }, ttsModel: { key: 'Text-to-Speech Model', }, }, ``` This configuration allows you to add and configure custom models and API keys, including those for external APIs like Litellm <sup>[[1]](https://github.com/langgenius/dify/blob/main/web/i18n/fr-FR/common.ts)</sup>. For more detailed guidance on integrating third-party APIs, you can refer to the [Tool Integration Guide](https://github.com/langgenius/dify-docs/blob/main/en/guides/tools/tool-configuration/README.md) <sup>[[2]](https://github.com/langgenius/dify-docs/blob/main/en/guides/tools/tool-configuration/README.md)</sup>. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#4913
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?