langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-22 04:25:29 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Compatibility Issues with OpenAI API in Agent Mode #4232

New Issue
Closed
opened 2026-02-21 18:05:19 -05:00 by yindo · 2 comments
yindo commented 2026-02-21 18:05:19 -05:00
Owner
Copy Link
Copy Source

Originally created by @XiaoLey on GitHub (Jun 24, 2024).

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • Please do not modify this template :) and fill in all the required fields.

Dify version

0.6.11

Cloud or Self Hosted

Self Hosted (Source)

Steps to reproduce

Description:

I have spent considerable time tracking these issues. There are unresolved problems with the compatibility of the openai_api_compatible model provider with the Agent. These issues include:

  1. Responses containing formatting artifacts.
  2. Memory function incorrectly recalling assistant's replies.

I discovered these issues while using the gpt-4o model from the OpenRouter provider, which is implemented based on the openai_api_compatible model provider.

Snipaste_2024-06-18_17-29-14

  1. Use the gpt-4o model from the OpenRouter provider in Agent mode.
  2. Observe responses for formatting artifacts.
  3. Test memory function for recalling previous interactions.

✔️ Expected Behavior

The OpenRouter model provider should work seamlessly with the Agent, providing correctly formatted responses and accurately recalling past interactions.

❌ Actual Behavior

The model produces responses with formatting artifacts and fails to recall previous assistant replies accurately.

Originally created by @XiaoLey on GitHub (Jun 24, 2024). ### Self Checks - [X] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [X] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [X] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [X] Please do not modify this template :) and fill in all the required fields. ### Dify version 0.6.11 ### Cloud or Self Hosted Self Hosted (Source) ### Steps to reproduce **Description:** I have spent considerable time tracking these issues. There are unresolved problems with the compatibility of the `openai_api_compatible` model provider with the Agent. These issues include: 1. Responses containing formatting artifacts. 2. Memory function incorrectly recalling assistant's replies. I discovered these issues while using the `gpt-4o` model from the `OpenRouter` provider, which is implemented based on the `openai_api_compatible` model provider. ![Snipaste_2024-06-18_17-29-14](https://github.com/langgenius/dify/assets/46273173/e4645f9b-6548-4fed-9179-7275f06c5080) 1. Use the `gpt-4o` model from the `OpenRouter` provider in Agent mode. 2. Observe responses for formatting artifacts. 3. Test memory function for recalling previous interactions. ### ✔️ Expected Behavior The `OpenRouter` model provider should work seamlessly with the Agent, providing correctly formatted responses and accurately recalling past interactions. ### ❌ Actual Behavior The model produces responses with formatting artifacts and fails to recall previous assistant replies accurately.
yindo added the 🤖 feat:agent label 2026-02-21 18:05:19 -05:00
yindo closed this issue 2026-02-21 18:05:19 -05:00
yindo commented 2026-02-21 18:05:19 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Jun 24, 2024):

To address the compatibility issues with the gpt-4o model from the OpenRouter provider in Agent mode, you can take the following steps:

  1. Adjust the response_format Parameter:

    • This parameter allows you to specify the format that the model must output, which can help mitigate formatting artifacts.
    • You can set the response_format to either text or json_object depending on your needs.

    Example configuration:

    parameter_rules:
  - name: response_format
    default: text

    Or for JSON format:

    parameter_rules:
  - name: response_format
    default: json_object

  2. Adjust Memory Settings:

    • Increase the context_size and max_tokens_to_sample parameters to help the model retain more context, potentially improving recall accuracy.
    • Enable and adjust the window size in the memory settings to manage the amount of context the model retains.

    Example configuration for context_size and max_tokens_to_sample:

    - variable: context_size
  default: "4096"
- variable: max_tokens_to_sample
  default: "4096"

    Example code to enable and adjust the window size:

    const handleWindowEnabledChange = useCallback((enabled: boolean) => {
  const newPayload = produce(config.data || MEMORY_DEFAULT, (draft) => {
    if (!draft.window)
      draft.window = { enabled: false, size: 50 }

    draft.window.enabled = enabled
  })

  onChange(newPayload)
}, [config, onChange])

const handleWindowSizeChange = useCallback((size: number | string) => {
  const newPayload = produce(payload || MEMORY_DEFAULT, (draft) => {
    if (!draft.window)
      draft.window = { enabled: true, size: 50 }
    let limitedSize: null | string | number = size
    if (limitedSize === '') {
      limitedSize = null
    }
    else {
      limitedSize = parseInt(limitedSize as string, 10)
      if (isNaN(limitedSize))
        limitedSize = 50

      if (limitedSize < 1)
        limitedSize = 1

      if (limitedSize > 100)
        limitedSize = 100
    }

    draft.window.size = limitedSize as number
  })
  onChange(newPayload)
}, [payload, onChange])

  3. Serialization and Deserialization of prompt_messages:

    • Ensure that the jsonable_encoder function correctly handles the serialization and deserialization of prompt_messages to avoid formatting artifacts and memory recall issues.

    Relevant code snippet:

    try:
    text, usage, tool_call = self._invoke_llm(
        node_data_model=node_data.model,
        model_instance=model_instance,
        prompt_messages=prompt_messages,
        tools=prompt_message_tools,
        stop=model_config.stop,
    )
    process_data['usage'] = jsonable_encoder(usage)
    process_data['tool_call'] = jsonable_encoder(tool_call)
    process_data['llm_text'] = text
except Exception as e:
    return NodeRunResult(
        status=WorkflowNodeExecutionStatus.FAILED,
        inputs=inputs,
        process_data={},
        outputs={
            '__is_success': 0,
            '__reason': str(e)
        },
        error=str(e),
        metadata={}
    )

By implementing these adjustments, you should be able to resolve the formatting artifacts and improve the memory recall of the assistant's replies [1][2][3][4][5][6][7].

To continue talking to Dosu, mention @dosu.

@dosubot[bot] commented on GitHub (Jun 24, 2024): <!-- Answer --> To address the compatibility issues with the `gpt-4o` model from the `OpenRouter` provider in Agent mode, you can take the following steps: 1. **Adjust the `response_format` Parameter**: - This parameter allows you to specify the format that the model must output, which can help mitigate formatting artifacts. - You can set the `response_format` to either `text` or `json_object` depending on your needs. Example configuration: ```yaml parameter_rules: - name: response_format default: text ``` Or for JSON format: ```yaml parameter_rules: - name: response_format default: json_object ``` 2. **Adjust Memory Settings**: - Increase the `context_size` and `max_tokens_to_sample` parameters to help the model retain more context, potentially improving recall accuracy. - Enable and adjust the `window` size in the memory settings to manage the amount of context the model retains. Example configuration for `context_size` and `max_tokens_to_sample`: ```yaml - variable: context_size default: "4096" - variable: max_tokens_to_sample default: "4096" ``` Example code to enable and adjust the window size: ```tsx const handleWindowEnabledChange = useCallback((enabled: boolean) => { const newPayload = produce(config.data || MEMORY_DEFAULT, (draft) => { if (!draft.window) draft.window = { enabled: false, size: 50 } draft.window.enabled = enabled }) onChange(newPayload) }, [config, onChange]) const handleWindowSizeChange = useCallback((size: number | string) => { const newPayload = produce(payload || MEMORY_DEFAULT, (draft) => { if (!draft.window) draft.window = { enabled: true, size: 50 } let limitedSize: null | string | number = size if (limitedSize === '') { limitedSize = null } else { limitedSize = parseInt(limitedSize as string, 10) if (isNaN(limitedSize)) limitedSize = 50 if (limitedSize < 1) limitedSize = 1 if (limitedSize > 100) limitedSize = 100 } draft.window.size = limitedSize as number }) onChange(newPayload) }, [payload, onChange]) ``` 3. **Serialization and Deserialization of `prompt_messages`**: - Ensure that the `jsonable_encoder` function correctly handles the serialization and deserialization of `prompt_messages` to avoid formatting artifacts and memory recall issues. Relevant code snippet: ```python try: text, usage, tool_call = self._invoke_llm( node_data_model=node_data.model, model_instance=model_instance, prompt_messages=prompt_messages, tools=prompt_message_tools, stop=model_config.stop, ) process_data['usage'] = jsonable_encoder(usage) process_data['tool_call'] = jsonable_encoder(tool_call) process_data['llm_text'] = text except Exception as e: return NodeRunResult( status=WorkflowNodeExecutionStatus.FAILED, inputs=inputs, process_data={}, outputs={ '__is_success': 0, '__reason': str(e) }, error=str(e), metadata={} ) ``` By implementing these adjustments, you should be able to resolve the formatting artifacts and improve the memory recall of the assistant's replies <sup>[[1]](https://github.com/langgenius/dify/blob/main/api/core/model_runtime/model_providers/openai/llm/gpt-4o.yaml)</sup><sup>[[2]](https://github.com/langgenius/dify/blob/main/api/core/model_runtime/model_providers/openai/llm/gpt-4o-2024-05-13.yaml)</sup><sup>[[3]](https://github.com/langgenius/dify/blob/main/api/core/model_runtime/model_providers/openai/llm/gpt-4.yaml)</sup><sup>[[4]](https://github.com/langgenius/dify/blob/main/api/core/model_runtime/model_providers/openrouter/openrouter.yaml)</sup><sup>[[5]](https://github.com/langgenius/dify/blob/main/web/app/components/workflow/nodes/_base/components/memory-config.tsx)</sup><sup>[[6]](https://github.com/langgenius/dify/blob/main/api/core/workflow/nodes/parameter_extractor/parameter_extractor_node.py)</sup><sup>[[7]](https://github.com/langgenius/dify/blob/main/api/core/memory/token_buffer_memory.py)</sup>. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
yindo commented 2026-02-21 18:05:19 -05:00
Author
Owner
Copy Link
Copy Source

@XiaoLey commented on GitHub (Jun 24, 2024):

This issue appears to have been resolved by PR #5494, so I will be closing it now.

@XiaoLey commented on GitHub (Jun 24, 2024): This issue appears to have been resolved by PR #5494, so I will be closing it now.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🤖 feat:agent
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#4232
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?