Stream Langchain qa chain answer #99

New Issue

2026-02-15T19:15:48-05:00

yindo commented

2026-02-15 19:15:48 -05:00

Originally created by @thomassrour on GitHub (Aug 13, 2024).

Hello,

I would like to stream the answer of my langchain qa chain. Here is how I'm trying to do it in the pipe method (almost as in #141) :

    self.local_llm  = ChatOllama(model="llama3:70b", 
                   #format="json",
                    temperature=0,  
                    base_url="http://...:11434",
                    streaming=True,
                    keep_alive= -1,
                    callbacks=[StreamingStdOutCallbackHandler()]
                    )

   chain =  RetrievalQA.from_chain_type(
              llm=self.local_llm,
              retriever=retriever,
              return_source_documents=False,
              chain_type_kwargs={'prompt': prompt},
              verbose = True,
              input_key="question",
          )
  
    for chunk in chain.run(user_message):
        yield chunk

However, the words don't appear one by one as they should, instead I get large chunks of about 50 words at once. Any help would be much appreciated, thank you.

Originally created by @thomassrour on GitHub (Aug 13, 2024). Hello, I would like to stream the answer of my langchain qa chain. Here is how I'm trying to do it in the pipe method (almost as in #141) : self.local_llm = ChatOllama(model="llama3:70b", #format="json", temperature=0, base_url="http://...:11434", streaming=True, keep_alive= -1, callbacks=[StreamingStdOutCallbackHandler()] ) chain = RetrievalQA.from_chain_type( llm=self.local_llm, retriever=retriever, return_source_documents=False, chain_type_kwargs={'prompt': prompt}, verbose = True, input_key="question", ) for chunk in chain.run(user_message): yield chunk However, the words don't appear one by one as they should, instead I get large chunks of about 50 words at once. Any help would be much appreciated, thank you.

yindo commented

2026-02-15 19:15:49 -05:00