langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 20:15:31 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Agent usage is not counting prompt tokens #16361

New Issue
Closed
opened 2026-02-21 19:25:48 -05:00 by yindo · 5 comments
yindo commented 2026-02-21 19:25:48 -05:00
Owner
Copy Link
Copy Source

Originally created by @DavideDelbianco on GitHub (Aug 22, 2025).

Originally assigned to: @laipz8200 on GitHub.

Self Checks

  • I have read the Contributing Guide and Language Policy.
  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report, otherwise it will be closed.
  • 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

1.7.2

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

Add an agent node to a flow.
Create a very long prompt and at the end instruct the agent to only answer with "yes" or "no"

Prompt:

You are an advanced large language model designed to follow instructions with extreme precision. Your task is to carefully read and analyze this entire prompt in detail. Do not skip any portion, and consider every instruction as important. The following sections are intended to increase the overall length of the prompt so that it consumes a significant number of tokens. The purpose of this is to stress-test the system and evaluate whether the prompt tokens are being accurately counted by the orchestration layer of the application in which you are running.
Section 1: General Behavior Rules
Always comply with the primary constraint at the end of the instructions.
Never provide additional explanations in your final output.
Never attempt to shorten your reasoning or provide a summary at the end.
The output must strictly be one of two words: YES or NO.
Section 2: Irrelevant Context for Token Padding
This section exists only to increase the length of the prompt. It contains unrelated content that should not affect your final output:
The history of computing begins with mechanical devices such as the abacus and the Antikythera mechanism, progressing through Babbage’s Analytical Engine, Turing’s theoretical machines, and eventually the development of electronic digital computers in the mid-20th century.
The rise of personal computing in the 1970s and 1980s, led by companies like Apple, IBM, and Microsoft, brought computing into homes and businesses worldwide.
Modern computing spans from embedded microcontrollers to massive distributed cloud systems, powering everything from smartphones and cars to scientific research and artificial intelligence.
Artificial intelligence itself has evolved through waves of optimism and skepticism, from symbolic reasoning systems in the mid-20th century to modern deep learning approaches using massive datasets and computational resources.
Section 3: Additional Length with Redundancy
Repeat key statements for token padding:
You must only answer with YES or NO.
You must only answer with YES or NO.
You must only answer with YES or NO.
You must only answer with YES or NO.
You must only answer with YES or NO.
Section 4: The Final Constraint
Regardless of the question asked after this prompt, your final and only response must be strictly one of the following two words:
YES
NO
No punctuation, no additional words, no formatting, no explanation, no reasoning.

End of Prompt — Now Answer This Question:
Is 2 + 2 equal to 4?

The token usage reported will report everything as Completion Tokens, making the usage calculations in tools like Langfuse, etc.. wrong.

For example the above prompt resulted in: 521 tokens
Image

From OpenAI tokenizer I know the prompt was 479
This means that the GPT4o-mini cost was:
$0.15 per 1M input and $0.60 per 1M output
479 * 0.15 * 0.000001 = $0.00007185
33 * 0.60 * 0.000001 = $0.0000198
total = $0.00009165

vs 512 * 0.6 * 0.0000001 = 0.0003072

Image

In dify the total cost MAYBE is correct, but once the values are passed to tracing tools the counts will be wrong.

Image

✔️ Expected Behavior

Prompt tokens should be counted

❌ Actual Behavior

Prompt tokens are not counted

Originally created by @DavideDelbianco on GitHub (Aug 22, 2025). Originally assigned to: @laipz8200 on GitHub. ### Self Checks - [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542). - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report, otherwise it will be closed. - [x] 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：） - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version 1.7.2 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce Add an agent node to a flow. Create a very long prompt and at the end instruct the agent to only answer with "yes" or "no" ``` Prompt: You are an advanced large language model designed to follow instructions with extreme precision. Your task is to carefully read and analyze this entire prompt in detail. Do not skip any portion, and consider every instruction as important. The following sections are intended to increase the overall length of the prompt so that it consumes a significant number of tokens. The purpose of this is to stress-test the system and evaluate whether the prompt tokens are being accurately counted by the orchestration layer of the application in which you are running. Section 1: General Behavior Rules Always comply with the primary constraint at the end of the instructions. Never provide additional explanations in your final output. Never attempt to shorten your reasoning or provide a summary at the end. The output must strictly be one of two words: YES or NO. Section 2: Irrelevant Context for Token Padding This section exists only to increase the length of the prompt. It contains unrelated content that should not affect your final output: The history of computing begins with mechanical devices such as the abacus and the Antikythera mechanism, progressing through Babbage’s Analytical Engine, Turing’s theoretical machines, and eventually the development of electronic digital computers in the mid-20th century. The rise of personal computing in the 1970s and 1980s, led by companies like Apple, IBM, and Microsoft, brought computing into homes and businesses worldwide. Modern computing spans from embedded microcontrollers to massive distributed cloud systems, powering everything from smartphones and cars to scientific research and artificial intelligence. Artificial intelligence itself has evolved through waves of optimism and skepticism, from symbolic reasoning systems in the mid-20th century to modern deep learning approaches using massive datasets and computational resources. Section 3: Additional Length with Redundancy Repeat key statements for token padding: You must only answer with YES or NO. You must only answer with YES or NO. You must only answer with YES or NO. You must only answer with YES or NO. You must only answer with YES or NO. Section 4: The Final Constraint Regardless of the question asked after this prompt, your final and only response must be strictly one of the following two words: YES NO No punctuation, no additional words, no formatting, no explanation, no reasoning. End of Prompt — Now Answer This Question: Is 2 + 2 equal to 4? ``` The token usage reported will report everything as Completion Tokens, making the usage calculations in tools like Langfuse, etc.. wrong. For example the above prompt resulted in: 521 tokens <img width="549" height="688" alt="Image" src="https://github.com/user-attachments/assets/01cccd6c-db15-4534-9faa-66a6c2f71a5f" /> From OpenAI tokenizer I know the prompt was 479 This means that the GPT4o-mini cost was: $0.15 per 1M input and $0.60 per 1M output 479 * 0.15 * 0.000001 = $0.00007185 33 * 0.60 * 0.000001 = $0.0000198 total = $0.00009165 vs 512 * 0.6 * 0.0000001 = 0.0003072 <img width="812" height="686" alt="Image" src="https://github.com/user-attachments/assets/7f66f831-73eb-4be3-8d82-dffbc268baca" /> In dify the total cost MAYBE is correct, but once the values are passed to tracing tools the counts will be wrong. <img width="299" height="282" alt="Image" src="https://github.com/user-attachments/assets/9000f928-05ce-4280-9786-22cdf2396ca3" /> ### ✔️ Expected Behavior Prompt tokens should be counted ### ❌ Actual Behavior Prompt tokens are not counted
yindo added the 🐞 bug label 2026-02-21 19:25:48 -05:00
yindo closed this issue 2026-02-21 19:25:49 -05:00
yindo commented 2026-02-21 19:25:49 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Aug 22, 2025):

This is a known issue in Dify agent nodes: prompt and completion tokens are not always separated correctly, and sometimes all tokens are reported as completion tokens or as zero. This affects cost tracking and integrations with tracing tools like Langfuse. The root cause is that agent nodes rely on the agent strategy's execution metadata to provide token usage details, but if the strategy or plugin doesn't split prompt and completion tokens properly, Dify can't report them accurately. This problem is specific to agent nodes—LLM nodes do not have this issue—and has been acknowledged by maintainers but is not yet fully fixed as of v1.7.2. Fixing it would require changes in how agent strategies and runners track and propagate token usage data through the workflow system. For now, there is no reliable workaround except to ensure any custom agent strategies or plugins you use correctly report prompt and completion tokens in their outputs. You can follow related discussions and updates here and here.

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other  Chat with Dosu Join Discord Share on X

@dosubot[bot] commented on GitHub (Aug 22, 2025): <!-- Answer --> This is a known issue in Dify agent nodes: prompt and completion tokens are not always separated correctly, and sometimes all tokens are reported as completion tokens or as zero. This affects cost tracking and integrations with tracing tools like Langfuse. The root cause is that agent nodes rely on the agent strategy's execution metadata to provide token usage details, but if the strategy or plugin doesn't split prompt and completion tokens properly, Dify can't report them accurately. This problem is specific to agent nodes—LLM nodes do not have this issue—and has been acknowledged by maintainers but is not yet fully fixed as of v1.7.2. Fixing it would require changes in how agent strategies and runners track and propagate token usage data through the workflow system. For now, there is no reliable workaround except to ensure any custom agent strategies or plugins you use correctly report prompt and completion tokens in their outputs. You can follow related discussions and updates [here](https://github.com/langgenius/dify/issues/21782#issuecomment-3030286715) and [here](https://github.com/langgenius/dify/issues/20485#issuecomment-2921283470). <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/17c2919e-24fa-405e-8608-c7d4baf1ee80?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/17c2919e-24fa-405e-8608-c7d4baf1ee80?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/17c2919e-24fa-405e-8608-c7d4baf1ee80?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/17c2919e-24fa-405e-8608-c7d4baf1ee80?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/17c2919e-24fa-405e-8608-c7d4baf1ee80?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/17c2919e-24fa-405e-8608-c7d4baf1ee80?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/17c2919e-24fa-405e-8608-c7d4baf1ee80?feedback_type=other)</sup>&nbsp;&nbsp;[![Chat with Dosu](https://dosu.dev/dosu-chat-badge.svg)](https://app.dosu.dev/6088a4f4-073a-43f3-b3d4-4f06fb0c05a0/ask?utm_source=github)&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/24360)
yindo commented 2026-02-21 19:25:49 -05:00
Author
Owner
Copy Link
Copy Source

@laipz8200 commented on GitHub (Aug 22, 2025):

Looks like AgentNode isn't correctly distinguishing prompt_tokens and completion_tokens

@laipz8200 commented on GitHub (Aug 22, 2025): Looks like AgentNode isn't correctly distinguishing prompt_tokens and completion_tokens
yindo commented 2026-02-21 19:25:49 -05:00
Author
Owner
Copy Link
Copy Source

@laipz8200 commented on GitHub (Aug 22, 2025):

link to https://github.com/langgenius/dify-official-plugins/issues/1564

@laipz8200 commented on GitHub (Aug 22, 2025): link to https://github.com/langgenius/dify-official-plugins/issues/1564
yindo commented 2026-02-21 19:25:49 -05:00
Author
Owner
Copy Link
Copy Source

@DavideDelbianco commented on GitHub (Aug 22, 2025):

that was fast ❤️ @laipz8200

@DavideDelbianco commented on GitHub (Aug 22, 2025): that was fast ❤️ @laipz8200
yindo commented 2026-02-21 19:25:49 -05:00
Author
Owner
Copy Link
Copy Source

@jortegac commented on GitHub (Feb 14, 2026):

A fix for this has been submitted in #32330. Agent nodes now populate process_data with model info and usage, and all tracing providers (Langfuse, LangSmith, Opik, Weave) recognize them as generation-type operations with proper token/cost tracking.

@jortegac commented on GitHub (Feb 14, 2026): A fix for this has been submitted in #32330. Agent nodes now populate `process_data` with model info and usage, and all tracing providers (Langfuse, LangSmith, Opik, Weave) recognize them as generation-type operations with proper token/cost tracking.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#16361
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?