langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 20:15:31 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Langfuse, Langsmith, Opik wrong token count #12525

New Issue
Closed
opened 2026-02-21 19:07:49 -05:00 by yindo · 3 comments
yindo commented 2026-02-21 19:07:49 -05:00
Owner
Copy Link
Copy Source

Originally created by @DavideDelbianco on GitHub (Apr 3, 2025).

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:)
  • Please do not modify this template :) and fill in all the required fields.

Dify version

1.1.3

Cloud or Self Hosted

Cloud, Self Hosted (Docker)

Steps to reproduce

The token count for the Suggested Question traces is incorrect in Langfuse and missing in Langsmith / Opik

Other details

I suggest a fix at the root of the problem instead of repeating the logic to retrieve the answer/completion tokens in every tracing implementation: ops_trace_manager.py
The info is already available and can be propagated with the various TraceInfo objects:

Image

✔️ Expected Behavior

The Suggested actions are generated by LLM and should report the token count based on prompt tokens / completion tokens

❌ Actual Behavior

Langfuse sends as input and output the character count instead of the tokens
Image

Langsmith and Opik lack the info altogether

Originally created by @DavideDelbianco on GitHub (Apr 3, 2025). ### Self Checks - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:) - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version 1.1.3 ### Cloud or Self Hosted Cloud, Self Hosted (Docker) ### Steps to reproduce The token count for the Suggested Question traces is incorrect in Langfuse and missing in Langsmith / Opik ### Other details I suggest a fix at the root of the problem instead of repeating the logic to retrieve the answer/completion tokens in every tracing implementation: ops_trace_manager.py The info is already available and can be propagated with the various TraceInfo objects: ![Image](https://github.com/user-attachments/assets/32875f2f-823a-4e1d-a7d2-5a199892b5af) ### ✔️ Expected Behavior The Suggested actions are generated by LLM and should report the token count based on prompt tokens / completion tokens ### ❌ Actual Behavior Langfuse sends as input and output the character count instead of the tokens ![Image](https://github.com/user-attachments/assets/612d8db6-8ba9-45ef-b3dc-107689ebc7dd) Langsmith and Opik lack the info altogether
yindo added the 🐞 bug label 2026-02-21 19:07:49 -05:00
yindo closed this issue 2026-02-21 19:07:49 -05:00
yindo commented 2026-02-21 19:07:49 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Apr 3, 2025):

The issue you're encountering seems to be related to incorrect token counting in Langfuse and missing token count information in Langsmith and Opik. The ops_trace_manager.py file is responsible for managing tracing instances and could be a central place to address this issue [1].

To resolve the problem, you might consider updating the ops_trace_manager.py to ensure that token counts are correctly calculated and propagated. The SuggestedQuestionTraceInfo object, which tracks suggested questions, could be modified to include accurate token counts instead of character counts [2]. This would involve adjusting the logic to calculate total_tokens using message_data.message_tokens and message_data.answer_tokens, as suggested in your report.

For Langfuse, ensure that the GenerationUsage object uses token counts instead of character counts by setting the appropriate attributes (promptTokens, completionTokens) and using UnitEnum.TOKENS instead of UnitEnum.CHARACTERS [3].

For Langsmith and Opik, you may need to implement similar logic to capture and propagate token count information, ensuring consistency across all tracing implementations.

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other

@dosubot[bot] commented on GitHub (Apr 3, 2025): <!-- Answer --> The issue you're encountering seems to be related to incorrect token counting in Langfuse and missing token count information in Langsmith and Opik. The `ops_trace_manager.py` file is responsible for managing tracing instances and could be a central place to address this issue <sup>[[1]](https://github.com/langgenius/dify/blob/main/api/core/ops/ops_trace_manager.py)</sup>. To resolve the problem, you might consider updating the `ops_trace_manager.py` to ensure that token counts are correctly calculated and propagated. The `SuggestedQuestionTraceInfo` object, which tracks suggested questions, could be modified to include accurate token counts instead of character counts <sup>[[2]](https://github.com/langgenius/dify-docs/blob/main/en/guides/monitoring/integrate-external-ops-tools/integrate-opik.md)</sup>. This would involve adjusting the logic to calculate `total_tokens` using `message_data.message_tokens` and `message_data.answer_tokens`, as suggested in your report. For Langfuse, ensure that the `GenerationUsage` object uses token counts instead of character counts by setting the appropriate attributes (`promptTokens`, `completionTokens`) and using `UnitEnum.TOKENS` instead of `UnitEnum.CHARACTERS` <sup>[[3]](https://github.com/langgenius/dify/blob/main/api/core/ops/langfuse_trace/entities/langfuse_trace_entity.py)</sup>. For Langsmith and Opik, you may need to implement similar logic to capture and propagate token count information, ensuring consistency across all tracing implementations. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/53f2b9a1-8c38-43de-9b6f-e4db79753322?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/53f2b9a1-8c38-43de-9b6f-e4db79753322?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/53f2b9a1-8c38-43de-9b6f-e4db79753322?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/53f2b9a1-8c38-43de-9b6f-e4db79753322?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/53f2b9a1-8c38-43de-9b6f-e4db79753322?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/53f2b9a1-8c38-43de-9b6f-e4db79753322?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/53f2b9a1-8c38-43de-9b6f-e4db79753322?feedback_type=other)</sup>
yindo commented 2026-02-21 19:07:49 -05:00
Author
Owner
Copy Link
Copy Source

@crazywoola commented on GitHub (Apr 3, 2025):

@lzs0603 PLTA

@crazywoola commented on GitHub (Apr 3, 2025): @lzs0603 PLTA
yindo commented 2026-02-21 19:07:49 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (May 4, 2025):

Hi, @DavideDelbianco. I'm Dosu, and I'm helping the Dify team manage their backlog. I'm marking this issue as stale.

Issue Summary:

  • You reported a bug with incorrect or missing token counts in Suggested Question traces across Langfuse, Langsmith, and Opik.
  • Langfuse uses character counts instead of token counts; Langsmith and Opik lack token information.
  • I suggested centralizing logic in ops_trace_manager.py for accurate token reporting.
  • Crazywoola tagged lzs0603 for further assistance or review.

Next Steps:

  • Please confirm if this issue is still relevant to the latest version of the Dify repository by commenting here.
  • If no updates are provided, the issue will be automatically closed in 15 days.

Thank you for your understanding and contribution!

@dosubot[bot] commented on GitHub (May 4, 2025): Hi, @DavideDelbianco. I'm [Dosu](https://dosu.dev), and I'm helping the Dify team manage their backlog. I'm marking this issue as stale. **Issue Summary:** - You reported a bug with incorrect or missing token counts in Suggested Question traces across Langfuse, Langsmith, and Opik. - Langfuse uses character counts instead of token counts; Langsmith and Opik lack token information. - I suggested centralizing logic in `ops_trace_manager.py` for accurate token reporting. - Crazywoola tagged lzs0603 for further assistance or review. **Next Steps:** - Please confirm if this issue is still relevant to the latest version of the Dify repository by commenting here. - If no updates are provided, the issue will be automatically closed in 15 days. Thank you for your understanding and contribution!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#12525
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?