mirror of
https://github.com/langchain-ai/langserve.git
synced 2026-07-19 22:03:48 -04:00
include_callback_events=True doesn't work in stream / stream_log endpoint #90
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @JayAhn2 on GitHub (Jan 8, 2024).
Originally assigned to: @eyurtsev on GitHub.
Hi!
I found that include_callback_events doesn't work in stream / stream_log.
I need to check token counts when I call stream / stream_log.
Is it intended?
Many thanks :)
@eyurtsev commented on GitHub (Jan 8, 2024):
Haven't gotten around to implementing it yet. Are you relying on RemoteRunnable for handling the callbacks?
Would sending the callbacks at the end of the stream work for your use case?
@JayAhn2 commented on GitHub (Jan 8, 2024):
@eyurtsev
Yes. I need to see callback_events at the end of the stream :)
It is required because my service only uses stream_log endpoint, and need to store token counts on each request.
@eyurtsev commented on GitHub (Jan 8, 2024):
it's end of stream vs. sending the callbacks in "real time" as they occur -- so they will be intermingled with event data. At the end shouldn't be too difficult to do.
End of stream shouldn't be too difficult to do, so i'll try to sequence that soonish
@JayAhn2 commented on GitHub (Jan 8, 2024):
@eyurtsev
IMHO, I guess real time callback isn't so meaningful, as callbacks are required for auditing the prompt and response for my case ;)
@eyurtsev commented on GitHub (Feb 1, 2024):
Hi @JayAhn2, I haven't picked up work on this yet
Quick question -- have you confirmed that you can get token counts when streaming with the model provider that you use?
I suspect that including callback events for streaming might not actually solve your use case as you'll need to run code client side to tokenize the response and count the tokens manually (even with callbacks included)