stream API returns entire answer after a while, instead of actually streaming the answer for ConversationalRetrievalChain #38

New Issue

Open

opened 2026-02-16 00:18:13 -05:00 by yindo · 17 comments

yindo commented 2026-02-16 00:18:13 -05:00

Owner

Copy Link

Copy Source

Originally created by @tigerinus on GitHub (Nov 13, 2023).

Originally assigned to: @eyurtsev on GitHub.

Hello,

I built a simple langchain app using ConversationalRetrievalChain and langserve.

It is working great for its invoke API. However when it comes to stream API, it returns entire answer after a while, instead of actually streaming the answer.

Here is the langserve part:

    streamer = TextStreamer(tokenizer=tokenizer, skip_prompt=True)

    pipeline = pipeline(
        "text-generation",
        model=model,
        tokenizer=tokenizer,
        max_length=1000,
        temperature=0.1,
        top_p=0.95,
        top_k=40,
        repetition_penalty=1.1,
        do_sample=True,
        streamer=streamer,
        # device=0,  # -1 for CPU
    )

    tokenizer.pad_token_id = model.config.eos_token_id

    llm = HuggingFacePipeline(
        pipeline=pipeline,
    )

    condenseQuestionPrompt = PromptTemplate.from_template(CONDENSE_QUESTION_TEMPLATE)

    prompt = PromptTemplate(template=TEMPLATE, input_variables=["context", "question"])
    chain = ConversationalRetrievalChain.from_llm(
        llm=llm,
        retriever=retriever,
        verbose=True,
        condense_question_prompt=condenseQuestionPrompt,
        combine_docs_chain_kwargs={"prompt": prompt},
        response_if_no_docs_found="I'm afraid that I don't have an answer to the question.",
    ).with_types(input_type=ChatHistory)

    app = FastAPI(
        title="app",
        version="0.1",
        description="The AI which knows everything about XYZ products.",
    )

    app.add_middleware(
        GZipMiddleware,
        minimum_size=1024,
    )

    app.add_middleware(
        CORSMiddleware,
        allow_origins=["*"],
        allow_methods=["*"],
        allow_headers=["*"],
    )

    add_routes(app, chain)

    uvicorn.run(app, host="0.0.0.0", port=8000)

Please help. Thanks!

Originally created by @tigerinus on GitHub (Nov 13, 2023). Originally assigned to: @eyurtsev on GitHub. Hello, I built a simple langchain app using `ConversationalRetrievalChain` and `langserve`. It is working great for its `invoke` API. However when it comes to `stream` API, it returns entire answer after a while, instead of actually streaming the answer. Here is the `langserve` part: ```python streamer = TextStreamer(tokenizer=tokenizer, skip_prompt=True) pipeline = pipeline( "text-generation", model=model, tokenizer=tokenizer, max_length=1000, temperature=0.1, top_p=0.95, top_k=40, repetition_penalty=1.1, do_sample=True, streamer=streamer, # device=0, # -1 for CPU ) tokenizer.pad_token_id = model.config.eos_token_id llm = HuggingFacePipeline( pipeline=pipeline, ) condenseQuestionPrompt = PromptTemplate.from_template(CONDENSE_QUESTION_TEMPLATE) prompt = PromptTemplate(template=TEMPLATE, input_variables=["context", "question"]) chain = ConversationalRetrievalChain.from_llm( llm=llm, retriever=retriever, verbose=True, condense_question_prompt=condenseQuestionPrompt, combine_docs_chain_kwargs={"prompt": prompt}, response_if_no_docs_found="I'm afraid that I don't have an answer to the question.", ).with_types(input_type=ChatHistory) app = FastAPI( title="app", version="0.1", description="The AI which knows everything about XYZ products.", ) app.add_middleware( GZipMiddleware, minimum_size=1024, ) app.add_middleware( CORSMiddleware, allow_origins=["*"], allow_methods=["*"], allow_headers=["*"], ) add_routes(app, chain) uvicorn.run(app, host="0.0.0.0", port=8000) ``` Please help. Thanks!

yindo commented 2026-02-16 00:18:14 -05:00

Author

Owner

Copy Link

Copy Source

@eyurtsev commented on GitHub (Nov 13, 2023):

@tigerinus could you confirm that this is a langserve issue rather than langchain issue? i..e,

   chain = ConversationalRetrievalChain.from_llm(
        llm=llm,
        retriever=retriever,
        verbose=True,
        condense_question_prompt=condenseQuestionPrompt,
        combine_docs_chain_kwargs={"prompt": prompt},
        response_if_no_docs_found="I'm afraid that I don't have an answer to the question.",
    ).with_types(input_type=ChatHistory)


for chunk in chain.stream(..): # <-- Does this work
  ...

@eyurtsev commented on GitHub (Nov 13, 2023): @tigerinus could you confirm that this is a langserve issue rather than langchain issue? i..e, ``` chain = ConversationalRetrievalChain.from_llm( llm=llm, retriever=retriever, verbose=True, condense_question_prompt=condenseQuestionPrompt, combine_docs_chain_kwargs={"prompt": prompt}, response_if_no_docs_found="I'm afraid that I don't have an answer to the question.", ).with_types(input_type=ChatHistory) for chunk in chain.stream(..): # <-- Does this work ... ```

yindo commented 2026-02-16 00:18:14 -05:00

Author

Owner

Copy Link

Copy Source

@tigerinus commented on GitHub (Nov 13, 2023):

@tigerinus could you confirm that this is a langserve issue rather than langchain issue? i..e,

   chain = ConversationalRetrievalChain.from_llm(
        llm=llm,
        retriever=retriever,
        verbose=True,
        condense_question_prompt=condenseQuestionPrompt,
        combine_docs_chain_kwargs={"prompt": prompt},
        response_if_no_docs_found="I'm afraid that I don't have an answer to the question.",
    ).with_types(input_type=ChatHistory)


for chunk in chain.stream(..): # <-- Does this work
  ...

Yes, it works. It prints the answer token by token.

    for chunk in chain.stream({"question": "What's capital city of Canada?", "chat_history": []}):
        print(chunk)

@tigerinus commented on GitHub (Nov 13, 2023): > @tigerinus could you confirm that this is a langserve issue rather than langchain issue? i..e, > > ``` > chain = ConversationalRetrievalChain.from_llm( > llm=llm, > retriever=retriever, > verbose=True, > condense_question_prompt=condenseQuestionPrompt, > combine_docs_chain_kwargs={"prompt": prompt}, > response_if_no_docs_found="I'm afraid that I don't have an answer to the question.", > ).with_types(input_type=ChatHistory) > > > for chunk in chain.stream(..): # <-- Does this work > ... > ``` Yes, it works. It prints the answer token by token. ```python for chunk in chain.stream({"question": "What's capital city of Canada?", "chat_history": []}): print(chunk) ```

yindo commented 2026-02-16 00:18:14 -05:00

Author

Owner

Copy Link

Copy Source

@eyurtsev commented on GitHub (Nov 14, 2023):

Could you check what happens if you remove the middleware for gzip compression?

@eyurtsev commented on GitHub (Nov 14, 2023): Could you check what happens if you remove the middleware for gzip compression?

yindo commented 2026-02-16 00:18:14 -05:00

Author

Owner

Copy Link

Copy Source

@tigerinus commented on GitHub (Nov 14, 2023):

Could you check what happens if you remove the middleware for gzip compression?

That didn't help.

How exactly should I call the /stream endpoint?

Does it work elsewhere?

@tigerinus commented on GitHub (Nov 14, 2023): > Could you check what happens if you remove the middleware for gzip compression? That didn't help. How exactly should I call the `/stream` endpoint? Does it work elsewhere?

yindo commented 2026-02-16 00:18:14 -05:00

Author

Owner

Copy Link

Copy Source

@eyurtsev commented on GitHub (Nov 14, 2023):

Using the RemoteClient. The RemoteClient should be pointing at the path of the runnable (not the /stream endpoint)

See example here in the middle that streams: https://github.com/langchain-ai/langserve/blob/main/examples/llm/client.ipynb

The server should stream if the underlying runnable used by the server can stream. You can also try astream_log from the RemoteClient

@eyurtsev commented on GitHub (Nov 14, 2023): Using the RemoteClient. The RemoteClient should be pointing at the path of the runnable (not the `/stream` endpoint) See example here in the middle that streams: https://github.com/langchain-ai/langserve/blob/main/examples/llm/client.ipynb The server should stream if the underlying runnable used by the server can stream. You can also try `astream_log` from the RemoteClient

yindo commented 2026-02-16 00:18:14 -05:00

Author

Owner

Copy Link

Copy Source

@tigerinus commented on GitHub (Nov 14, 2023):

I will try that notebook.

However our frontend is written in VueJS. Would be nice to have an example code in JS.

If we call /stream directly we see the answer returned in whole chunk

@tigerinus commented on GitHub (Nov 14, 2023): I will try that notebook. However our frontend is written in VueJS. Would be nice to have an example code in JS. If we call `/stream` directly we see the answer returned in whole chunk 

yindo commented 2026-02-16 00:18:14 -05:00

Author

Owner

Copy Link

Copy Source

@tigerinus commented on GitHub (Nov 15, 2023):

@eyurtsev I have no idea how to get the notebook example working for my app:

I tried

from langchain.prompts.chat import ChatPromptTemplate

from langserve import RemoteRunnable
llm = RemoteRunnable('http://localhost:8000')

prompt = ChatPromptTemplate.from_messages({
    "input": {
        "question": "Hi",
        "chat_history": []
    }
}).format_messages()

llm.invoke(prompt)

But the backend always say something like:

INFO:     127.0.0.1:52946 - "POST /invoke HTTP/1.1" 500 Internal Server Error                                                                                                                               
ERROR:    Exception in ASGI application                                                                                                                                                                     
Traceback (most recent call last):                                                                                                                                                                          
  File "/home/tiger/dev/IceWhaleTech/IceWhale-101-GPT/llm_app_langserve/venv/lib/python3.11/site-packages/pydantic/v1/main.py", line 716, in validate                                                       
    value_as_dict = dict(value)                                                                                                                                                                             
                    ^^^^^^^^^^^                                                                                                                                                                             
ValueError: dictionary update sequence element #0 has length 4; 2 is required                                                                                                                               

@tigerinus commented on GitHub (Nov 15, 2023): @eyurtsev I have no idea how to get the notebook example working for my app: I tried ```python from langchain.prompts.chat import ChatPromptTemplate from langserve import RemoteRunnable llm = RemoteRunnable('http://localhost:8000') prompt = ChatPromptTemplate.from_messages({ "input": { "question": "Hi", "chat_history": [] } }).format_messages() llm.invoke(prompt) ``` But the backend always say something like: ``` INFO: 127.0.0.1:52946 - "POST /invoke HTTP/1.1" 500 Internal Server Error ERROR: Exception in ASGI application Traceback (most recent call last): File "/home/tiger/dev/IceWhaleTech/IceWhale-101-GPT/llm_app_langserve/venv/lib/python3.11/site-packages/pydantic/v1/main.py", line 716, in validate value_as_dict = dict(value) ^^^^^^^^^^^ ValueError: dictionary update sequence element #0 has length 4; 2 is required ```

yindo commented 2026-02-16 00:18:14 -05:00

Author

Owner

Copy Link

Copy Source

@eyurtsev commented on GitHub (Nov 15, 2023):

for js: Have you tried the js remote runnable client? https://api.js.langchain.com/classes/runnables_remote.RemoteRunnable.html

absent any bugs, it should be a drop in replacement -- so you could launch the server code provided in the example, make confirm the python client code works, and then replace with js version

For your code snippet could you include both server and client code?

In the example I referenced the remote runnables take a path to the mounted location of the runnable:

openai_llm = RemoteRunnable("http://localhost:8000/openai/")
anthropic = RemoteRunnable("http://localhost:8000/anthropic/")

@eyurtsev commented on GitHub (Nov 15, 2023): for js: Have you tried the js remote runnable client? https://api.js.langchain.com/classes/runnables_remote.RemoteRunnable.html absent any bugs, it should be a drop in replacement -- so you could launch the server code provided in the example, make confirm the python client code works, and then replace with js version For your code snippet could you include both server and client code? In the example I referenced the remote runnables take a path to the mounted location of the runnable: ``` openai_llm = RemoteRunnable("http://localhost:8000/openai/") anthropic = RemoteRunnable("http://localhost:8000/anthropic/") ```

yindo commented 2026-02-16 00:18:14 -05:00

Author

Owner

Copy Link

Copy Source

@gabegaz commented on GitHub (Nov 15, 2023):

I also encountered the same issue

@gabegaz commented on GitHub (Nov 15, 2023): I also encountered the same issue

yindo commented 2026-02-16 00:18:15 -05:00

Author

Owner

Copy Link

Copy Source

@eyurtsev commented on GitHub (Nov 15, 2023):

@gabegaz do you have a snippet of both client and server code you're using? If i have full code, I can try to recreate locally

@eyurtsev commented on GitHub (Nov 15, 2023): @gabegaz do you have a snippet of both client and server code you're using? If i have full code, I can try to recreate locally

yindo commented 2026-02-16 00:18:15 -05:00

Author

Owner

Copy Link

Copy Source

@gabegaz commented on GitHub (Nov 15, 2023):

Here is my server side code snippet:

`from langchain.llms import LlamaCpp
from langchain.callbacks.manager import CallbackManager
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
from fastapi import FastAPI
from langserve import add_routes

n_gpu_layers = 1
n_batch = 512
callback_manager = CallbackManager([StreamingStdOutCallbackHandler()])

llm = LlamaCpp(
model_path=mistral+"mistral-7b-instruct-v0.1.Q5_K_M.gguf",
n_gpu_layers=n_gpu_layers,
n_batch=n_batch,
n_ctx=2048,
f16_kv=True,
callback_manager=callback_manager,
verbose=True,
)

app = FastAPI(
title="LangChain Server",
version="1.0",
description="A simple api server using Langchain's Runnable interfaces",
)

add_routes(
app,
llm,
path="/datalem",
)

if name == "main":
import uvicorn
uvicorn.run(app, host="localhost", port=8000)`

And here is the client code:

`from langchain.prompts.chat import ChatPromptTemplate
from langserve import RemoteRunnable
from .models import Chat
from django.http import JsonResponse
from django.utils import timezone
from django.shortcuts import render

datalem = RemoteRunnable("http://localhost:5000/datalem/")

def chatbot_langserve(request):
chats = Chat.objects.filter(user=request.user.id)

if request.method == 'POST':
    message = request.POST.get('message')
    
    prompt = ChatPromptTemplate.from_messages(
         [
              (
                   "system",
                   "You are a highly educated person who loves to use big words. "
                   + "You are also concise. Never answer in more than three sentences.",                       ),
                   ("human", message),
                   ]).format_messages()

    # response =datalem.invoke(prompt)
    for chunk in datalem.stream(prompt):
        response=(chunk.content, end="", flush=True)

    chat = Chat(user=request.user, message=message,
                response=response, created_at=timezone.now)
    chat.save()
    return JsonResponse({'message': message, 'response': response})
return render(request, 'chatbot.html', {'chats': chats})

My problem is on the client side. How can I get response from this print() function?
# response =datalem.invoke(prompt) for chunk in datalem.stream(prompt): print(chunk.content, end="", flush=True)

@gabegaz commented on GitHub (Nov 15, 2023): Here is my server side code snippet: `from langchain.llms import LlamaCpp from langchain.callbacks.manager import CallbackManager from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler from fastapi import FastAPI from langserve import add_routes n_gpu_layers = 1 n_batch = 512 callback_manager = CallbackManager([StreamingStdOutCallbackHandler()]) llm = LlamaCpp( model_path=mistral+"mistral-7b-instruct-v0.1.Q5_K_M.gguf", n_gpu_layers=n_gpu_layers, n_batch=n_batch, n_ctx=2048, f16_kv=True, callback_manager=callback_manager, verbose=True, ) app = FastAPI( title="LangChain Server", version="1.0", description="A simple api server using Langchain's Runnable interfaces", ) add_routes( app, llm, path="/datalem", ) if __name__ == "__main__": import uvicorn uvicorn.run(app, host="localhost", port=8000)` And here is the client code: `from langchain.prompts.chat import ChatPromptTemplate from langserve import RemoteRunnable from .models import Chat from django.http import JsonResponse from django.utils import timezone from django.shortcuts import render datalem = RemoteRunnable("http://localhost:5000/datalem/") def chatbot_langserve(request): chats = Chat.objects.filter(user=request.user.id) if request.method == 'POST': message = request.POST.get('message') prompt = ChatPromptTemplate.from_messages( [ ( "system", "You are a highly educated person who loves to use big words. " + "You are also concise. Never answer in more than three sentences.", ), ("human", message), ]).format_messages() # response =datalem.invoke(prompt) for chunk in datalem.stream(prompt): response=(chunk.content, end="", flush=True) chat = Chat(user=request.user, message=message, response=response, created_at=timezone.now) chat.save() return JsonResponse({'message': message, 'response': response}) return render(request, 'chatbot.html', {'chats': chats}) My problem is on the client side. How can I get response from this print() function? ` # response =datalem.invoke(prompt) for chunk in datalem.stream(prompt): print(chunk.content, end="", flush=True)`

yindo commented 2026-02-16 00:18:15 -05:00

Author

Owner

Copy Link

Copy Source

@eyurtsev commented on GitHub (Nov 16, 2023):

@tigerinus , @gabegaz Thanks for the code. The underlying chain (ConversationalRetrievalChain) does not support streaming, so this is not a langserve issue.

TLDR:

Use this for conversational retriever: https://python.langchain.com/docs/expression_language/cookbook/retrieval#conversational-retrieval-chain

How to debug whether streaming is supported

The best way to debug is use the server chain (without any server in the middle) and confirm whether it streams or not.

In this case, you can test in the following manner:

from langchain.chains import ConversationalRetrievalChain
from langchain.chat_models import ChatOpenAI
from langchain.embeddings import OpenAIEmbeddings
from langchain.pydantic_v1 import BaseModel, Field
from langchain.vectorstores import FAISS

vectorstore = FAISS.from_texts(
    ["cats like fish", "dogs like sticks"], embedding=OpenAIEmbeddings()
)
retriever = vectorstore.as_retriever()
model = ChatOpenAI()
chain = ConversationalRetrievalChain.from_llm(model, retriever)


# User input
class ChatHistory(BaseModel):
    """Chat history with the bot."""

    chat_history: List[Tuple[str, str]] = Field(
        ...,
        extra={"widget": {"type": "chat", "input": "question", "output": "answer"}},
    )
    question: str


chain = ConversationalRetrievalChain.from_llm(model, retriever).with_types(
    input_type=ChatHistory
)

async for chunk in chain.astream(
    {"question": "tell me along story about bears", "chat_history": [("hi", "hi")]}
):
    print(chunk)

You'll see that the underlying chain only has one chunk! Sot it definitely does not stream.

If you create it using LCEL, confirm first that streaming works server side and then hook it into langserve

@eyurtsev commented on GitHub (Nov 16, 2023): @tigerinus , @gabegaz Thanks for the code. The underlying chain (`ConversationalRetrievalChain`) does not support streaming, so this is not a langserve issue. TLDR: Use this for conversational retriever: https://python.langchain.com/docs/expression_language/cookbook/retrieval#conversational-retrieval-chain ## How to debug whether streaming is supported The best way to debug is use the server chain (without any server in the middle) and confirm whether it streams or not. In this case, you can test in the following manner: ```from typing import List, Tuple from langchain.chains import ConversationalRetrievalChain from langchain.chat_models import ChatOpenAI from langchain.embeddings import OpenAIEmbeddings from langchain.pydantic_v1 import BaseModel, Field from langchain.vectorstores import FAISS vectorstore = FAISS.from_texts( ["cats like fish", "dogs like sticks"], embedding=OpenAIEmbeddings() ) retriever = vectorstore.as_retriever() model = ChatOpenAI() chain = ConversationalRetrievalChain.from_llm(model, retriever) # User input class ChatHistory(BaseModel): """Chat history with the bot.""" chat_history: List[Tuple[str, str]] = Field( ..., extra={"widget": {"type": "chat", "input": "question", "output": "answer"}}, ) question: str chain = ConversationalRetrievalChain.from_llm(model, retriever).with_types( input_type=ChatHistory ) async for chunk in chain.astream( {"question": "tell me along story about bears", "chat_history": [("hi", "hi")]} ): print(chunk) ``` You'll see that the underlying chain only has one chunk! Sot it definitely does not stream. If you create it using LCEL, confirm first that streaming works server side and then hook it into langserve

yindo commented 2026-02-16 00:18:15 -05:00

Author

Owner

Copy Link

Copy Source

@eyurtsev commented on GitHub (Nov 16, 2023):

Updated the example: https://github.com/langchain-ai/langserve/pull/238

See this code: https://github.com/langchain-ai/langserve/blob/main/examples/conversational_retrieval_chain/server.py
and corresponding client: https://github.com/langchain-ai/langserve/blob/main/examples/conversational_retrieval_chain/client.ipynb

@eyurtsev commented on GitHub (Nov 16, 2023): Updated the example: https://github.com/langchain-ai/langserve/pull/238 See this code: https://github.com/langchain-ai/langserve/blob/main/examples/conversational_retrieval_chain/server.py and corresponding client: https://github.com/langchain-ai/langserve/blob/main/examples/conversational_retrieval_chain/client.ipynb

yindo commented 2026-02-16 00:18:15 -05:00

Author

Owner

Copy Link

Copy Source

@tigerinus commented on GitHub (Nov 16, 2023):

It is by design or a bug that ConversationalRetrievalChain doesn't support streaming?

@tigerinus commented on GitHub (Nov 16, 2023): It is by design or a bug that ConversationalRetrievalChain doesn't support streaming?

yindo commented 2026-02-16 00:18:15 -05:00

Author

Owner

Copy Link

Copy Source

@eyurtsev commented on GitHub (Nov 16, 2023):

Neither/both?

ConversationalRetrievalChain is an old style langchain chain that doesn't fully utilize LCEL.

A lot of the older objects do not support all the capabilities that LCEL offers (e.g., native async, optimized batching, stream, stream log etc.)

We're not deprecating the old style objects since users depend on them, but we're also not enhancing them in many cases since it's hard and also doesn't really make sense since LCEL a lot more flexibility to customize the behavior properly for the given application domain.

@eyurtsev commented on GitHub (Nov 16, 2023): Neither/both? ConversationalRetrievalChain is an old style langchain chain that doesn't fully utilize LCEL. A lot of the older objects do not support all the capabilities that LCEL offers (e.g., native async, optimized batching, stream, stream log etc.) We're not deprecating the old style objects since users depend on them, but we're also not enhancing them in many cases since it's hard and also doesn't really make sense since LCEL a lot more flexibility to customize the behavior properly for the given application domain.

yindo commented 2026-02-16 00:18:15 -05:00

Author

Owner

Copy Link

Copy Source

@rickknowles-cognitant commented on GitHub (Jan 4, 2024):

Thanks @eyurtsev - the previous reply was super helpful to understand the nature of the problem I've been seeing (which is pretty similar to the OP's problem).

My question is then: is there something we can do as a workaround ? It appears that Chainlit has done something async using a callback handler that is effectively granting them the ability to do it.

(image taken from https://docs.chainlit.io/api-reference/integrations/langchain )

The source for the handler appears here: https://github.com/Chainlit/chainlit/blob/main/backend/chainlit/langchain/callbacks.py

it's then invoked in chainlit message calls as:

From what I can see in the source (although I'm not sure if I'm reading it right) they are using a combination of a Tracer subclass and consuming the entire thing to pull out the element they want ?

Is this an advisable approach ? Or is there something else you'd recommend to people who wanted to get to this sort of functionality in a langserve environment ?

Thanks in advance,

Rick

@rickknowles-cognitant commented on GitHub (Jan 4, 2024): Thanks @eyurtsev - the previous reply was super helpful to understand the nature of the problem I've been seeing (which is pretty similar to the OP's problem). My question is then: is there something we can do as a workaround ? It appears that Chainlit has done something async using a callback handler that is effectively granting them the ability to do it.  (image taken from https://docs.chainlit.io/api-reference/integrations/langchain ) The source for the handler appears here: https://github.com/Chainlit/chainlit/blob/main/backend/chainlit/langchain/callbacks.py it's then invoked in chainlit message calls as:  From what I can see in the source (although I'm not sure if I'm reading it right) they are using a combination of a Tracer subclass and consuming the entire thing to pull out the element they want ? Is this an advisable approach ? Or is there something else you'd recommend to people who wanted to get to this sort of functionality in a langserve environment ? Thanks in advance, Rick

yindo commented 2026-02-16 00:18:15 -05:00

Author

Owner

Copy Link

Copy Source

@mhadi4194 commented on GitHub (Feb 11, 2024):

> My problem is on the client side. How can I get response from this print() function? ` # response =datalem.invoke(prompt) for chunk in datalem.stream(prompt): print(chunk.content, end="", flush=True)`

@gabegaz As you mentioned, the problem seems to be on client side. As client receives the tokens, it is supposed to render the html or update the user interface (which is typically done in other languages such as js).

see this example that I got from googling: "how to render streaming response in python on client side?"

instead of this:

if request.method == 'POST':
    message = request.POST.get('message')
    
    prompt = ChatPromptTemplate.from_messages(
         [
              (
                   "system",
                   "You are a highly educated person who loves to use big words. "
                   + "You are also concise. Never answer in more than three sentences.",                       ),
                   ("human", message),
                   ]).format_messages()

    # response =datalem.invoke(prompt)
    for chunk in datalem.stream(prompt):
        response=(chunk.content, end="", flush=True)

    chat = Chat(user=request.user, message=message,
                response=response, created_at=timezone.now)
    chat.save()
    return JsonResponse({'message': message, 'response': response})
return render(request, 'chatbot.html', {'chats': chats})

execute something like this code on client side:

from django.http import HttpResponse
datalem = RemoteRunnable("http://localhost:5000/datalem/")


def chatbot_langserve(request):

    # removing save chat to keep it simple
#     chats = Chat.objects.filter(user=request.user.id)

    if request.method == 'POST':
        message = request.POST.get('message')

#        message = input()  # typically client sends requests to server. I'm not sure about your set up, it appears that "client" is
processing post request instead of receiving user input form UI /console ?


        prompt = ChatPromptTemplate.from_messages(
             [
                  (
                       "system",
                       "You are a highly educated person who loves to use big words. "
                       + "You are also concise. Never answer in more than three sentences.",                       ),
                       ("human", message),
                       ]).format_messages()


        def generate():
            for chunk in datalem.stream(prompt):
                yield chunk.content
    
    return HttpResponse(generate(), content_type='text/plain')
    

@mhadi4194 commented on GitHub (Feb 11, 2024): ``` > My problem is on the client side. How can I get response from this print() function? ` # response =datalem.invoke(prompt) for chunk in datalem.stream(prompt): print(chunk.content, end="", flush=True)` ``` @gabegaz As you mentioned, the problem seems to be on client side. As client receives the tokens, it is supposed to render the html or update the user interface (which is typically done in other languages such as js). see this [example](https://dev.to/anoff/how-to-create-a-streaming-http-interface-in-python--ceh) that I got from googling: "how to render streaming response in python on client side?" instead of this: ``` if request.method == 'POST': message = request.POST.get('message') prompt = ChatPromptTemplate.from_messages( [ ( "system", "You are a highly educated person who loves to use big words. " + "You are also concise. Never answer in more than three sentences.", ), ("human", message), ]).format_messages() # response =datalem.invoke(prompt) for chunk in datalem.stream(prompt): response=(chunk.content, end="", flush=True) chat = Chat(user=request.user, message=message, response=response, created_at=timezone.now) chat.save() return JsonResponse({'message': message, 'response': response}) return render(request, 'chatbot.html', {'chats': chats}) ``` execute something like this code on client side: ``` from django.http import HttpResponse datalem = RemoteRunnable("http://localhost:5000/datalem/") def chatbot_langserve(request): # removing save chat to keep it simple # chats = Chat.objects.filter(user=request.user.id) if request.method == 'POST': message = request.POST.get('message') # message = input() # typically client sends requests to server. I'm not sure about your set up, it appears that "client" is processing post request instead of receiving user input form UI /console ? prompt = ChatPromptTemplate.from_messages( [ ( "system", "You are a highly educated person who loves to use big words. " + "You are also concise. Never answer in more than three sentences.", ), ("human", message), ]).format_messages() def generate(): for chunk in datalem.stream(prompt): yield chunk.content return HttpResponse(generate(), content_type='text/plain') ```

yindo referenced this issue 2026-02-16 00:20:04 -05:00

[PR #38] [CLOSED] Add langsmith quick start in README #294

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

dependabot/pip/pip-82fdfb001f

dependabot/pip/jupyterlab-4.5.6

dependabot/pip/fastapi-0.136.0

dependabot/pip/pydantic-2.13.3

dependabot/pip/sse-starlette-3.3.4

dependabot/pip/pytest-asyncio-1.3.0

sam/bump-langchain

joaquin/test-pushing-sk-to-langserver-should-block

joaquin/bump-again-due-to-failed-release

eyurtsev-patch-5

wfh/relax_sse_starlette

eyurtsev-patch-4

jacob/docs

nc/22apr/serve

add-playground-test

eugene/playground_security

eyurtsev-patch-3

jacob/chat_widget

eyurtsev-patch-2

eyurtsev-patch-1

eugene/add_is_instance_check

eugene/agent_executor_playground

revert-209-2023-11-10-format-langsmith-runs

2023-11-10-format-langsmith-runs

harrison/dont-do-validation

nc/4nov

eugene/playground

eugene/exmaples

dqbd/playground-autotype-tuple-fix

erick/packages-bugfix

erick/assign-input-params

jacob/playground

jacob/allowlist

erick/langservehub

eugene/add_interpreptors

v0.3.3

v0.3.2

v0.3.1

v0.3.0

v0.3.0dev2

v0.3.0dev1

v0.2.3

v0.2.2

v0.2.1

v0.2.0

v0.2.0rc1

v0.1.1

v0.1.0

v0.1.0rc2

v0.1.0rc1

v0.0.51

v0.0.50

v0.0.49

v0.0.48

v0.0.47

v0.0.46

v0.0.45

v0.0.44

v0.0.43

v0.0.42

v0.0.41

v0.0.40

v0.0.39

v0.0.38

v0.0.37

v0.0.36

v0.0.35

v0.0.34

v0.0.33

v0.0.32

v0.0.31

v0.0.30

v0.0.29

v0.0.28

v0.0.27

v0.0.26

v0.0.25

v0.0.24

v0.0.23

v0.0.22

v0.0.21

v0.0.20

v0.0.19

v0.0.18

v0.0.17

v0.0.16

v0.0.15

v0.0.14

v0.0.13

v0.0.12

v0.0.11

v0.0.10

v0.0.9

v0.0.8

v0.0.7

v0.0.6

v0.0.5

v0.0.4

v0.0.3

v0.0.2

Labels

Clear labels

bug

confirmed

duplicate

enhancement

help wanted

in-progress

investigate

not langserve

playground

pull-request

Mirrored from GitHub Pull Request

question

stale

streaming

unable-to-reproduce

wontfix

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

yindo

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: langchain-ai/langserve#38