langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 20:15:31 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

The message_tokens data in the messages table is abnormal. #8329

New Issue
Closed
opened 2026-02-21 18:25:06 -05:00 by yindo · 3 comments
yindo commented 2026-02-21 18:25:06 -05:00
Owner
Copy Link
Copy Source

Originally created by @majestichou on GitHub (Feb 17, 2025).

Originally assigned to: @vgudur297 on GitHub.

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

0.15.3

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

  1. I use the following DSL to create an app and this app has memory.
app:
  description: ''
  icon: 🤖
  icon_background: '#FFEAD5'
  mode: advanced-chat
  name: deepseek-qwen-llama-70b
  use_icon_as_answer_icon: false
kind: app
version: 0.1.5
workflow:
  conversation_variables: []
  environment_variables: []
  features:
    file_upload:
      allowed_file_extensions:
      - .JPG
      - .JPEG
      - .PNG
      - .GIF
      - .WEBP
      - .SVG
      allowed_file_types:
      - image
      allowed_file_upload_methods:
      - local_file
      - remote_url
      enabled: false
      fileUploadConfig:
        audio_file_size_limit: 50
        batch_count_limit: 5
        file_size_limit: 15
        image_file_size_limit: 10
        video_file_size_limit: 100
        workflow_file_upload_limit: 10
      image:
        enabled: false
        number_limits: 3
        transfer_methods:
        - local_file
        - remote_url
      number_limits: 3
    opening_statement: ''
    retriever_resource:
      enabled: true
    sensitive_word_avoidance:
      enabled: false
    speech_to_text:
      enabled: false
    suggested_questions: []
    suggested_questions_after_answer:
      enabled: false
    text_to_speech:
      enabled: false
      language: ''
      voice: ''
  graph:
    edges:
    - data:
        sourceType: start
        targetType: llm
      id: 1739773269234-llm
      source: '1739773269234'
      sourceHandle: source
      target: llm
      targetHandle: target
      type: custom
    - data:
        sourceType: llm
        targetType: answer
      id: llm-answer
      source: llm
      sourceHandle: source
      target: answer
      targetHandle: target
      type: custom
    nodes:
    - data:
        desc: ''
        selected: false
        title: 开始
        type: start
        variables: []
      height: 54
      id: '1739773269234'
      position:
        x: 80
        y: 282
      positionAbsolute:
        x: 80
        y: 282
      sourcePosition: right
      targetPosition: left
      type: custom
      width: 244
    - data:
        context:
          enabled: false
          variable_selector: []
        desc: ''
        memory:
          query_prompt_template: '{{#sys.query#}}'
          role_prefix:
            assistant: ''
            user: ''
          window:
            enabled: true
            size: 10
        model:
          completion_params:
            max_tokens: 4096
            temperature: 0.6
            top_p: 0.8
          mode: chat
          name: /data/models/DeepSeek-R1-Distill-Llama-70B
          provider: openai_api_compatible
        prompt_template:
        - id: ace60b14-ea7c-4239-a9d2-ed0c8ef15506
          role: system
          text: ''
        selected: false
        title: LLM
        type: llm
        variables: []
        vision:
          enabled: false
      height: 98
      id: llm
      position:
        x: 380
        y: 282
      positionAbsolute:
        x: 380
        y: 282
      selected: true
      sourcePosition: right
      targetPosition: left
      type: custom
      width: 244
    - data:
        answer: '{{#llm.text#}}'
        desc: ''
        selected: false
        title: 直接回复
        type: answer
        variables: []
      height: 103
      id: answer
      position:
        x: 680
        y: 282
      positionAbsolute:
        x: 680
        y: 282
      selected: false
      sourcePosition: right
      targetPosition: left
      type: custom
      width: 244
    viewport:
      x: -150
      y: 115
      zoom: 1
  1. I asked two questions in this app .
    The first is "为什么1+1等于2", the second is "帮我写一个单例模式，采用python". The models all give better answers.
  2. I want to know how many tokens were consumed in the first conversation and how many tokens were consumed in the second conversation. So I connected to dify's database and check the messages data table. The data is displayed as follows：
query message message_tokens
为什么1+1等于2 """""" 15
帮我写一个单例模式，采用python """""" 15

This data is obviously abnormal.
The app is memory-based, and when the second question is asked, the first question and answer record is brought in, and the second question is stitched together and sent to the big model. Therefore, the second message_tokens should be much larger than the first. But now they're the same size. What's even more strange is that the message data is empty（ """"""） for both questions.

✔️ Expected Behavior

query message message_tokens
为什么1+1等于2 为什么1+1等于2 15
帮我写一个单例模式，采用python Q: 为什么1+1等于2？A: 1+1等于2的原因可以从多个角度进行解释：日常生活的直观理解：1代表一个单位，2代表两个单位。当我们将一个单位与另一个单位合并时，总共有两个单位，因此1+1=2。数学公理系统的定义：在皮亚诺公理等数学公理系统中，1是一个基本单位，而加法运算通过递归定义，2被定义为1的后继数，因此1+1=2。逻辑推理的证明：在数学基础理论中，如《数学原理》中，1+1=2通过一系列公理和定理进行详细证明，涉及数论和集合论等高级数学领域。不同数学体系中的运算规则：在某些数学体系中，如模2运算，1+1可能等于0。然而，在我们日常使用的十进制系统中，1+1=2是一个基本且恒成立的等式。综上所述，1+1等于2是因为它反映了数量的增加和组合，由数学公理系统和逻辑推理所定义和证明，并且在日常使用的十进制系统中是一个基本而重要的事实。Q: 帮我写一个单例模式，采用python 1500

❌ Actual Behavior

query message message_tokens
为什么1+1等于2 """""" 15
帮我写一个单例模式，采用python """""" 15
Originally created by @majestichou on GitHub (Feb 17, 2025). Originally assigned to: @vgudur297 on GitHub. ### Self Checks - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:） - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version 0.15.3 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce 1. I use the following DSL to create an app and this app has memory. ``` app: description: '' icon: 🤖 icon_background: '#FFEAD5' mode: advanced-chat name: deepseek-qwen-llama-70b use_icon_as_answer_icon: false kind: app version: 0.1.5 workflow: conversation_variables: [] environment_variables: [] features: file_upload: allowed_file_extensions: - .JPG - .JPEG - .PNG - .GIF - .WEBP - .SVG allowed_file_types: - image allowed_file_upload_methods: - local_file - remote_url enabled: false fileUploadConfig: audio_file_size_limit: 50 batch_count_limit: 5 file_size_limit: 15 image_file_size_limit: 10 video_file_size_limit: 100 workflow_file_upload_limit: 10 image: enabled: false number_limits: 3 transfer_methods: - local_file - remote_url number_limits: 3 opening_statement: '' retriever_resource: enabled: true sensitive_word_avoidance: enabled: false speech_to_text: enabled: false suggested_questions: [] suggested_questions_after_answer: enabled: false text_to_speech: enabled: false language: '' voice: '' graph: edges: - data: sourceType: start targetType: llm id: 1739773269234-llm source: '1739773269234' sourceHandle: source target: llm targetHandle: target type: custom - data: sourceType: llm targetType: answer id: llm-answer source: llm sourceHandle: source target: answer targetHandle: target type: custom nodes: - data: desc: '' selected: false title: 开始 type: start variables: [] height: 54 id: '1739773269234' position: x: 80 y: 282 positionAbsolute: x: 80 y: 282 sourcePosition: right targetPosition: left type: custom width: 244 - data: context: enabled: false variable_selector: [] desc: '' memory: query_prompt_template: '{{#sys.query#}}' role_prefix: assistant: '' user: '' window: enabled: true size: 10 model: completion_params: max_tokens: 4096 temperature: 0.6 top_p: 0.8 mode: chat name: /data/models/DeepSeek-R1-Distill-Llama-70B provider: openai_api_compatible prompt_template: - id: ace60b14-ea7c-4239-a9d2-ed0c8ef15506 role: system text: '' selected: false title: LLM type: llm variables: [] vision: enabled: false height: 98 id: llm position: x: 380 y: 282 positionAbsolute: x: 380 y: 282 selected: true sourcePosition: right targetPosition: left type: custom width: 244 - data: answer: '{{#llm.text#}}' desc: '' selected: false title: 直接回复 type: answer variables: [] height: 103 id: answer position: x: 680 y: 282 positionAbsolute: x: 680 y: 282 selected: false sourcePosition: right targetPosition: left type: custom width: 244 viewport: x: -150 y: 115 zoom: 1 ``` 2. I asked two questions in this app . The first is "为什么1+1等于2", the second is "帮我写一个单例模式，采用python". The models all give better answers. 3. I want to know how many tokens were consumed in the first conversation and how many tokens were consumed in the second conversation. So I connected to dify's database and check the messages data table. The data is displayed as follows： | query | message | message_tokens | |----|----|----| | 为什么1+1等于2 | """""" | 15 | | 帮我写一个单例模式，采用python | """""" | 15 | **This data is obviously abnormal.** The app is memory-based, and when the second question is asked, the first question and answer record is brought in, and the second question is stitched together and sent to the big model. Therefore, the second message_tokens should be much larger than the first. But now they're the same size. What's even more strange is that the message data is empty（ """"""） for both questions. ### ✔️ Expected Behavior | query | message | message_tokens | |----|----|----| | 为什么1+1等于2 | 为什么1+1等于2 | 15 | | 帮我写一个单例模式，采用python |Q: 为什么1+1等于2？A: 1+1等于2的原因可以从多个角度进行解释：日常生活的直观理解：1代表一个单位，2代表两个单位。当我们将一个单位与另一个单位合并时，总共有两个单位，因此1+1=2。数学公理系统的定义：在皮亚诺公理等数学公理系统中，1是一个基本单位，而加法运算通过递归定义，2被定义为1的后继数，因此1+1=2。逻辑推理的证明：在数学基础理论中，如《数学原理》中，1+1=2通过一系列公理和定理进行详细证明，涉及数论和集合论等高级数学领域。不同数学体系中的运算规则：在某些数学体系中，如模2运算，1+1可能等于0。然而，在我们日常使用的十进制系统中，1+1=2是一个基本且恒成立的等式。综上所述，1+1等于2是因为它反映了数量的增加和组合，由数学公理系统和逻辑推理所定义和证明，并且在日常使用的十进制系统中是一个基本而重要的事实。Q: 帮我写一个单例模式，采用python | 1500 | ### ❌ Actual Behavior | query | message | message_tokens | |----|----|----| | 为什么1+1等于2 | """""" | 15 | | 帮我写一个单例模式，采用python | """""" | 15 |
yindo added the 🐞 buggood first issuestaleoutdated labels 2026-02-21 18:25:06 -05:00
yindo closed this issue 2026-02-21 18:25:06 -05:00
yindo commented 2026-02-21 18:25:06 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Feb 17, 2025):

The issue you're experiencing with the message_tokens and empty message fields in the messages table could be related to how tool calls are handled in the code. When a message involves tool calls, the Content is set to a space (" ") to avoid issues with certain SDKs, and in some cases, the content is explicitly set to an empty string to prevent displaying content on the web page when tool calls are present [1].

Additionally, the logic for storing messages and their corresponding tokens is defined in the Message class within the api/models/model.py file. The message field stores the message content in JSON format, and message_tokens stores the number of tokens associated with the message [2].

If the app is memory-based, the token calculation should consider the context of previous interactions. The function _handle_memory_completion_mode manages memory in completion models and calculates the remaining tokens available for the prompt, which might not be reflected correctly in your current setup [3][4].

You might want to check if the logic for handling memory and token calculation is correctly implemented and if there are any specific conditions or configurations affecting the storage of messages and tokens.

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other

@dosubot[bot] commented on GitHub (Feb 17, 2025): <!-- Answer --> The issue you're experiencing with the `message_tokens` and empty `message` fields in the messages table could be related to how tool calls are handled in the code. When a message involves tool calls, the `Content` is set to a space (" ") to avoid issues with certain SDKs, and in some cases, the `content` is explicitly set to an empty string to prevent displaying content on the web page when tool calls are present <sup>[[1]](https://github.com/langgenius/dify/blob/main/api/core/model_runtime/model_providers/hunyuan/llm/llm.py)</sup>. Additionally, the logic for storing messages and their corresponding tokens is defined in the `Message` class within the `api/models/model.py` file. The `message` field stores the message content in JSON format, and `message_tokens` stores the number of tokens associated with the message <sup>[[2]](https://github.com/langgenius/dify/blob/main/api/models/model.py)</sup>. If the app is memory-based, the token calculation should consider the context of previous interactions. The function `_handle_memory_completion_mode` manages memory in completion models and calculates the remaining tokens available for the prompt, which might not be reflected correctly in your current setup <sup>[[3]](https://github.com/langgenius/dify/blob/main/api/core/workflow/nodes/llm/node.py)</sup><sup>[[4]](https://github.com/langgenius/dify/blob/main/api/core/app/apps/base_app_runner.py)</sup>. You might want to check if the logic for handling memory and token calculation is correctly implemented and if there are any specific conditions or configurations affecting the storage of messages and tokens. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/a68d9bcf-bacc-4453-a03b-89c8b085c6b7?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/a68d9bcf-bacc-4453-a03b-89c8b085c6b7?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/a68d9bcf-bacc-4453-a03b-89c8b085c6b7?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/a68d9bcf-bacc-4453-a03b-89c8b085c6b7?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/a68d9bcf-bacc-4453-a03b-89c8b085c6b7?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/a68d9bcf-bacc-4453-a03b-89c8b085c6b7?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/a68d9bcf-bacc-4453-a03b-89c8b085c6b7?feedback_type=other)</sup>
yindo commented 2026-02-21 18:25:07 -05:00
Author
Owner
Copy Link
Copy Source

@vgudur297 commented on GitHub (Mar 30, 2025):

@majestichou , I'll work on this if no one else is working

@vgudur297

@vgudur297 commented on GitHub (Mar 30, 2025): @majestichou , I'll work on this if no one else is working @vgudur297
yindo commented 2026-02-21 18:25:07 -05:00
Author
Owner
Copy Link
Copy Source

@crazywoola commented on GitHub (Apr 8, 2025):

@vgudur297 Feel free to try it out.

@crazywoola commented on GitHub (Apr 8, 2025): @vgudur297 Feel free to try it out.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label good first issue outdated stale 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#8329
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?