langchain-ai/langserve
1
0
Fork 0
mirror of https://github.com/langchain-ai/langserve.git synced 2026-07-18 10:54:29 -04:00
Code Issues 136 Packages Projects Releases 10 Wiki Activity

What is the best way to get token usage via api response? #110

New Issue
Closed
opened 2026-02-16 00:18:51 -05:00 by yindo · 4 comments
yindo commented 2026-02-16 00:18:51 -05:00
Owner
Copy Link
Copy Source

Originally created by @teeppp on GitHub (Jan 30, 2024).

I want to get the token usage to display token count in frontend application.
I found it has already included in callback_events when include_callback_event=True.
But, there are too much information.
is there how to exclude unnessesary infomartions from callback_events, or other way？

Originally created by @teeppp on GitHub (Jan 30, 2024). I want to get the token usage to display token count in frontend application. I found it has already included in callback_events when include_callback_event=True. But, there are too much information. is there how to exclude unnessesary infomartions from callback_events, or other way？
yindo closed this issue 2026-02-16 00:18:51 -05:00
yindo commented 2026-02-16 00:18:51 -05:00
Author
Owner
Copy Link
Copy Source

@eyurtsev commented on GitHub (Feb 1, 2024):

If you're using the remote runnable client with invoke use:

https://python.langchain.com/docs/modules/callbacks/token_counting

If you're doing it with your own custom code, then you'll need to filter until you encounter the appropriate callback event that corresponds to on llm end.

Keep in mind that:

  1. callbacks are not considered stable yet, they're part of the API, but I haven't tested extensively the callbacks so not sure how well they work
  2. callbacks are not included with streaming/stream_log/stream_events at the moment
  3. If you're using open AI: open AI does not include token counts with streamin (if I'm not mistaken), so callbacks wouldn't help (at least not unless the token counting code above is modified to work with streaming response from open ai).
  4. If you're using other model providers, you will need to check if there's a way to count tokens with them.

One simple thing you could do:

  1. Get the response on the client side
  2. Use a tokenizer to estimate the number of tokens
@eyurtsev commented on GitHub (Feb 1, 2024): If you're using the remote runnable client with invoke use: https://python.langchain.com/docs/modules/callbacks/token_counting If you're doing it with your own custom code, then you'll need to filter until you encounter the appropriate callback event that corresponds to on llm end. Keep in mind that: 1. callbacks are not considered stable yet, they're part of the API, but I haven't tested extensively the callbacks so not sure how well they work 2. callbacks are not included with streaming/stream_log/stream_events at the moment 3. If you're using open AI: open AI does not include token counts with streamin (if I'm not mistaken), so callbacks wouldn't help (at least not unless the token counting code above is modified to work with streaming response from open ai). 4. If you're using other model providers, you will need to check if there's a way to count tokens with them. One simple thing you could do: 1. Get the response on the client side 2. Use a tokenizer to estimate the number of tokens
yindo commented 2026-02-16 00:18:51 -05:00
Author
Owner
Copy Link
Copy Source

@teeppp commented on GitHub (Feb 2, 2024):

@eyurtsev
Thank you for your reply. I understood it.
as you point it out, my application is not use the runnable client.
And, the your simple solution to use tokenizer on clientside is not good way for me because we have some intermediate steps depends on RAG. there are some internal prompts only in the backend application. this case, we cannot estimate it on client side accurately.

But, fortunately, my current application not use streaming for now. i will use callback as a short term solution.
As a future request, i hope it to implement to easily to some additional information to api response if possible.

@teeppp commented on GitHub (Feb 2, 2024): @eyurtsev Thank you for your reply. I understood it. as you point it out, my application is not use the runnable client. And, the your simple solution to use tokenizer on clientside is not good way for me because we have some intermediate steps depends on RAG. there are some internal prompts only in the backend application. this case, we cannot estimate it on client side accurately. But, fortunately, my current application not use streaming for now. i will use callback as a short term solution. As a future request, i hope it to implement to easily to some additional information to api response if possible.
yindo commented 2026-02-16 00:18:51 -05:00
Author
Owner
Copy Link
Copy Source

@eyurtsev commented on GitHub (Feb 5, 2024):

One option is to do the calculation on the server side and return it as part of the response:


from langchain_openai.chat_models import ChatOpenAI
from langchain_core.prompts import PromptTemplate
from langchain_core.runnables import RunnableMap, RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser

prompt = PromptTemplate.from_template("{input}")

def calculate_tokens(text: str) -> int:
    return len(text) # Replace with appropriate tokenization logic

model = ChatOpenAI()

chain = prompt | RunnableMap({
    "output": model,
    "input_tokens": lambda prompt: calculate_tokens(prompt.text)
}) | RunnablePassthrough.assign(output_tokens=lambda x: calculate_tokens(x['output'].content))

chain.invoke({'input': 'hello'})
@eyurtsev commented on GitHub (Feb 5, 2024): One option is to do the calculation on the server side and return it as part of the response: ```python from langchain_openai.chat_models import ChatOpenAI from langchain_core.prompts import PromptTemplate from langchain_core.runnables import RunnableMap, RunnablePassthrough from langchain_core.output_parsers import StrOutputParser prompt = PromptTemplate.from_template("{input}") def calculate_tokens(text: str) -> int: return len(text) # Replace with appropriate tokenization logic model = ChatOpenAI() chain = prompt | RunnableMap({ "output": model, "input_tokens": lambda prompt: calculate_tokens(prompt.text) }) | RunnablePassthrough.assign(output_tokens=lambda x: calculate_tokens(x['output'].content)) chain.invoke({'input': 'hello'}) ```
yindo commented 2026-02-16 00:18:51 -05:00
Author
Owner
Copy Link
Copy Source

@teeppp commented on GitHub (Feb 14, 2024):

thanks. it seems the option is better way than using callback infos.
i will try it.

@teeppp commented on GitHub (Feb 14, 2024): thanks. it seems the option is better way than using callback infos. i will try it.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
confirmed
duplicate
enhancement
help wanted
in-progress
investigate
not langserve
playground
pull-request
Mirrored from GitHub Pull Request
 question
stale
streaming
unable-to-reproduce
wontfix
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langchain-ai/langserve#110
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?