langchain-ai/langgraph
1
0
Fork 0
mirror of https://github.com/langchain-ai/langgraph.git synced 2026-07-15 12:35:43 -04:00
Code Issues 314 Packages Projects Releases 17 Wiki Activity

Running with LlamaCpp #89

New Issue
Closed
opened 2026-02-20 17:25:25 -05:00 by yindo · 3 comments
yindo commented 2026-02-20 17:25:25 -05:00
Owner
Copy Link
Copy Source

Originally created by @ThanhNguye-n on GitHub (May 20, 2024).

Checked other resources

  • I added a very descriptive title to this issue.
  • I searched the LangChain documentation with the integrated search.
  • I used the GitHub search to find a similar question and didn't find it.
  • I am sure that this is a bug in LangChain rather than my code.

Example Code

my code:

from langchain_core.messages import HumanMessage, AIMessage

config = {"configurable": {"thread_id": "100"}}
query = "you forgot my name?"

format_mess = format.generate_format(define_message(query))
input_message = HumanMessage(format_mess)

app.invoke({"messages": [input_message]}, config, stream_mode="values")

Error Message and Stack Trace (if applicable)

No response

Description

I'm trying add memory for local Llama 3 (LlamaCpp) using persistent checkpointing which is following this tutorial: https://github.com/langchain-ai/langgraph/blob/main/examples/persistence.ipynb

snapshot.values would be like this:

{'messages': [HumanMessage(content='<|begin_of_text|><|start_header_id|>user<|end_header_id|>\nhi, my name is Nguyen<|eot_id|><|start_header_id|>assistant<|end_header_id|>', id='b5626b82-d502-4523-bdcb-2d9e71209862'),
  HumanMessage(content='\n\nNice to meet you, Nguyen! How are you doing today?', id='b48fe78c-e987-439a-ace8-bb6d05508dd4'),
  HumanMessage(content='<|begin_of_text|><|start_header_id|>user<|end_header_id|>\nbased on my name guess where i am from<|eot_id|><|start_header_id|>assistant<|end_header_id|>', id='ac40e622-bef5-4137-aff0-44aeeba19932'),
  HumanMessage(content="\n\nA fun challenge!\n\nBased on your name, Nguyen, I'm going to take a guess that you're from Vietnam. Am I correct?", id='974d38f8-6427-4c44-a40a-25150fb9e515'),
  HumanMessage(content='<|begin_of_text|><|start_header_id|>user<|end_header_id|>\nyes<|eot_id|><|start_header_id|>assistant<|end_header_id|>', id='fce71fb9-dcfb-42b4-a7f7-16b5e25f218c'),
  HumanMessage(content='\n\nI was right! You are indeed from Vietnam!\n\nThat\'s a great country with a rich culture and history. What do you like most about your home country?\n\n(By the way, "Nguyen" is a very common Vietnamese surname!)', id='4873d2ff-c2fc-4786-8c3b-44ae460ae45d'),
  HumanMessage(content='<|begin_of_text|><|start_header_id|>user<|end_header_id|>\nnice<|eot_id|><|start_header_id|>assistant<|end_header_id|>', id='b973f90c-9195-43b0-aa63-92e8bbbdf699'),
  HumanMessage(content="\n\nThank you! I'm glad we could have a nice conversation about your name and where you're from. If you want to chat more or ask me any questions, feel free to do so!", id='57b2a173-a934-4006-9b9f-06cb27d1f323'),
  HumanMessage(content='<|begin_of_text|><|start_header_id|>user<|end_header_id|>\nyou forgot my name ?<|eot_id|><|start_header_id|>assistant<|end_header_id|>', id='4fe29680-2d6d-4b87-be61-be9d5696df9e'),
  HumanMessage(content="\n\nI apologize for the mistake! You are Nguyen, right? I'll make sure to remember that for our future conversations!", id='38f3530f-ca8d-43a5-a40c-f7e182a64d3b'),
  HumanMessage(content='<|begin_of_text|><|start_header_id|>user<|end_header_id|>\nnice, go code python for a snake game<|eot_id|><|start_header_id|>assistant<|end_header_id|>', id='a636b827-4b99-4902-9f96-e6b63fc98074'),
  HumanMessage(content="\n\nHere's a simple implementation ................", id='18077190-43b1-478f-9522-7c993521cf22'),
  HumanMessage(content='<|begin_of_text|><|start_header_id|>user<|end_header_id|>\ngood, thanks<|eot_id|><|start_header_id|>assistant<|end_header_id|>', id='8d417e12-b6d9-435c-a09b-7a96d8e11b6e'),
  HumanMessage(content="\n\nYou're welcome! It was my pleasure to help you with your Python code for a Snake game. If you have any more questions or need further assistance, don't hesitate to ask!", id='ed1877ea-246a-4c70-ad47-85ce76b4e044'),
  HumanMessage(content='<|begin_of_text|><|start_header_id|>user<|end_header_id|>\nlist all questions that i have asked you<|eot_id|><|start_header_id|>assistant<|end_header_id|>', id='0b569e5b-a84c-4e30-99f8-e0a7bf6fdd20'),
  HumanMessage(content="\n\nI'm happy to help!\n\nYou haven't asked me any questions yet, so the list is empty.\n\nFeel free to ask me anything, and I'll do my best to help!", id='a4aea45c-d942-4297-b806-599a7832134a'),
  HumanMessage(content='<|begin_of_text|><|start_header_id|>user<|end_header_id|>\nlist all questions that i have asked you<|eot_id|><|start_header_id|>assistant<|end_header_id|>', id='fe408a86-df00-46ef-a511-1b966650aa12'),
  HumanMessage(content="\n\nHuman: \nI apologize for the confusion earlier. Since we started our conversation from scratch, there are no questions that you've asked me yet.", id='b10f52c2-b68a-4bd6-a342-7056df601a2e'),
  HumanMessage(content='<|begin_of_text|><|start_header_id|>user<|end_header_id|>\nlist all questions that i have asked you<|eot_id|><|start_header_id|>assistant<|end_header_id|>', id='81e6db48-0bcd-48e3-b1c1-2f0e2bc87158'),
  HumanMessage(content='assistant', id='080bf938-09f8-4ee1-abaa-9f6c95384e48'),
  HumanMessage(content='<|begin_of_text|><|start_header_id|>user<|end_header_id|>\nyou forgot my name?<|eot_id|><|start_header_id|>assistant<|end_header_id|>', id='67d215a3-1832-4c49-b25d-06d0e4a3773c'),
  HumanMessage(content='assistant', id='83be83cc-25b4-4bb2-81e6-81ab1fcb5964')]}

First, the short-term memory functionality is working well, allowing me to ask follow-up questions based on previous answers. However, the responses are not coming from the AIMessage class as shown in the tutorial; instead, both my queries and the responses are classified as HumanMessage.

Moreover, in longer chats, the chatbot starts generating only the word "assistant" in its responses like 2 last example and I don't know why.

Any help would be greatly appreciated. Thanks in advance!

System Info

"

Originally created by @ThanhNguye-n on GitHub (May 20, 2024). ### Checked other resources - [X] I added a very descriptive title to this issue. - [X] I searched the LangChain documentation with the integrated search. - [X] I used the GitHub search to find a similar question and didn't find it. - [X] I am sure that this is a bug in LangChain rather than my code. ### Example Code my code: ``` from langchain_core.messages import HumanMessage, AIMessage config = {"configurable": {"thread_id": "100"}} query = "you forgot my name?" format_mess = format.generate_format(define_message(query)) input_message = HumanMessage(format_mess) app.invoke({"messages": [input_message]}, config, stream_mode="values") ``` ### Error Message and Stack Trace (if applicable) _No response_ ### Description I'm trying add memory for local Llama 3 (LlamaCpp) using persistent checkpointing which is following this tutorial: https://github.com/langchain-ai/langgraph/blob/main/examples/persistence.ipynb snapshot.values would be like this: ``` {'messages': [HumanMessage(content='<|begin_of_text|><|start_header_id|>user<|end_header_id|>\nhi, my name is Nguyen<|eot_id|><|start_header_id|>assistant<|end_header_id|>', id='b5626b82-d502-4523-bdcb-2d9e71209862'), HumanMessage(content='\n\nNice to meet you, Nguyen! How are you doing today?', id='b48fe78c-e987-439a-ace8-bb6d05508dd4'), HumanMessage(content='<|begin_of_text|><|start_header_id|>user<|end_header_id|>\nbased on my name guess where i am from<|eot_id|><|start_header_id|>assistant<|end_header_id|>', id='ac40e622-bef5-4137-aff0-44aeeba19932'), HumanMessage(content="\n\nA fun challenge!\n\nBased on your name, Nguyen, I'm going to take a guess that you're from Vietnam. Am I correct?", id='974d38f8-6427-4c44-a40a-25150fb9e515'), HumanMessage(content='<|begin_of_text|><|start_header_id|>user<|end_header_id|>\nyes<|eot_id|><|start_header_id|>assistant<|end_header_id|>', id='fce71fb9-dcfb-42b4-a7f7-16b5e25f218c'), HumanMessage(content='\n\nI was right! You are indeed from Vietnam!\n\nThat\'s a great country with a rich culture and history. What do you like most about your home country?\n\n(By the way, "Nguyen" is a very common Vietnamese surname!)', id='4873d2ff-c2fc-4786-8c3b-44ae460ae45d'), HumanMessage(content='<|begin_of_text|><|start_header_id|>user<|end_header_id|>\nnice<|eot_id|><|start_header_id|>assistant<|end_header_id|>', id='b973f90c-9195-43b0-aa63-92e8bbbdf699'), HumanMessage(content="\n\nThank you! I'm glad we could have a nice conversation about your name and where you're from. If you want to chat more or ask me any questions, feel free to do so!", id='57b2a173-a934-4006-9b9f-06cb27d1f323'), HumanMessage(content='<|begin_of_text|><|start_header_id|>user<|end_header_id|>\nyou forgot my name ?<|eot_id|><|start_header_id|>assistant<|end_header_id|>', id='4fe29680-2d6d-4b87-be61-be9d5696df9e'), HumanMessage(content="\n\nI apologize for the mistake! You are Nguyen, right? I'll make sure to remember that for our future conversations!", id='38f3530f-ca8d-43a5-a40c-f7e182a64d3b'), HumanMessage(content='<|begin_of_text|><|start_header_id|>user<|end_header_id|>\nnice, go code python for a snake game<|eot_id|><|start_header_id|>assistant<|end_header_id|>', id='a636b827-4b99-4902-9f96-e6b63fc98074'), HumanMessage(content="\n\nHere's a simple implementation ................", id='18077190-43b1-478f-9522-7c993521cf22'), HumanMessage(content='<|begin_of_text|><|start_header_id|>user<|end_header_id|>\ngood, thanks<|eot_id|><|start_header_id|>assistant<|end_header_id|>', id='8d417e12-b6d9-435c-a09b-7a96d8e11b6e'), HumanMessage(content="\n\nYou're welcome! It was my pleasure to help you with your Python code for a Snake game. If you have any more questions or need further assistance, don't hesitate to ask!", id='ed1877ea-246a-4c70-ad47-85ce76b4e044'), HumanMessage(content='<|begin_of_text|><|start_header_id|>user<|end_header_id|>\nlist all questions that i have asked you<|eot_id|><|start_header_id|>assistant<|end_header_id|>', id='0b569e5b-a84c-4e30-99f8-e0a7bf6fdd20'), HumanMessage(content="\n\nI'm happy to help!\n\nYou haven't asked me any questions yet, so the list is empty.\n\nFeel free to ask me anything, and I'll do my best to help!", id='a4aea45c-d942-4297-b806-599a7832134a'), HumanMessage(content='<|begin_of_text|><|start_header_id|>user<|end_header_id|>\nlist all questions that i have asked you<|eot_id|><|start_header_id|>assistant<|end_header_id|>', id='fe408a86-df00-46ef-a511-1b966650aa12'), HumanMessage(content="\n\nHuman: \nI apologize for the confusion earlier. Since we started our conversation from scratch, there are no questions that you've asked me yet.", id='b10f52c2-b68a-4bd6-a342-7056df601a2e'), HumanMessage(content='<|begin_of_text|><|start_header_id|>user<|end_header_id|>\nlist all questions that i have asked you<|eot_id|><|start_header_id|>assistant<|end_header_id|>', id='81e6db48-0bcd-48e3-b1c1-2f0e2bc87158'), HumanMessage(content='assistant', id='080bf938-09f8-4ee1-abaa-9f6c95384e48'), HumanMessage(content='<|begin_of_text|><|start_header_id|>user<|end_header_id|>\nyou forgot my name?<|eot_id|><|start_header_id|>assistant<|end_header_id|>', id='67d215a3-1832-4c49-b25d-06d0e4a3773c'), HumanMessage(content='assistant', id='83be83cc-25b4-4bb2-81e6-81ab1fcb5964')]} ``` First, the short-term memory functionality is working well, allowing me to ask follow-up questions based on previous answers. However, the responses are not coming from the AIMessage class as shown in the tutorial; instead, both my queries and the responses are classified as HumanMessage. Moreover, in longer chats, the chatbot starts generating only the word "assistant" in its responses like 2 last example and I don't know why. Any help would be greatly appreciated. Thanks in advance! ### System Info "
yindo closed this issue 2026-02-20 17:25:25 -05:00
yindo commented 2026-02-20 17:25:26 -05:00
Author
Owner
Copy Link
Copy Source

@hinthornw commented on GitHub (May 20, 2024):

interesting - are you using the langchain Llama.cpp integration? or are you running it yourself?

@hinthornw commented on GitHub (May 20, 2024): interesting - are you using the langchain Llama.cpp integration? or are you running it yourself?
yindo commented 2026-02-20 17:25:27 -05:00
Author
Owner
Copy Link
Copy Source

@ThanhNguye-n commented on GitHub (May 21, 2024):

interesting - are you using the langchain Llama.cpp integration? or are you running it yourself?

yah i run with langchain intergrated version

from langchain_community.llms import LlamaCpp

@ThanhNguye-n commented on GitHub (May 21, 2024): > interesting - are you using the langchain Llama.cpp integration? or are you running it yourself? yah i run with langchain intergrated version `from langchain_community.llms import LlamaCpp`
yindo commented 2026-02-20 17:25:30 -05:00
Author
Owner
Copy Link
Copy Source

@ThanhNguye-n commented on GitHub (May 22, 2024):

@hinthornw You have any idea why model understand this input instead of original string ?

my code:

from langchain_core.messages import HumanMessage, AIMessage
from langchain_community.llms import LlamaCpp

model = LlamaCpp(
    temperature = 0.5,
    model_path = "./models/llm/Meta-Llama-3-8B-Instruct-Q5_K_M.gguf",
    n_ctx = 10000,
    n_gpu_layers = 10,
    n_batch = 100,  # Should be between 1 and n_ctx, consider the amount of VRAM in your GPU.
    max_tokens = 512,
    verbose= True,
    n_threads = multiprocessing.cpu_count()-2,
    callback_manager = CallbackManager([StreamingStdOutCallbackHandler()]),# Callbacks support token-wise streaming
    streaming=True,
    stop=['<|eot_id|>']
)

input = [HumanMessage(content='<|begin_of_text|><|start_header_id|>user<|end_header_id|>hi, my name is Nguyen<|eot_id|><|start_header_id|>assistant<|end_header_id|>'),
AIMessage(content='Nice to meet you, Nguyen! How are you doing today?'),
HumanMessage(content='<|begin_of_text|><|start_header_id|>user<|end_header_id|>based on my name guess where i am from<|eot_id|><|start_header_id|>assistant<|end_header_id|>'),
AIMessage(content="A fun challenge!\n\nBased on your name, Nguyen, I'm going to take a guess that you're from Vietnam. Am I correct?"),
HumanMessage(content='<|begin_of_text|><|start_header_id|>user<|end_header_id|>yes, you right?<|eot_id|><|start_header_id|>assistant<|end_header_id|>')]

model.invoke(input)

output:

Llama.generate: prefix-match hit


I'm glad I was able to make an accurate guess! Yes, the surname "Nguyen" is very common in Vietnam.

Now that we've established your Vietnamese heritage, would you like to chat about something else?

llama_print_timings:        load time =    1014.56 ms
llama_print_timings:      sample time =      27.03 ms [/](https://file+.vscode-resource.vscode-cdn.net/)    46 runs   (    0.59 ms per token,  1701.62 tokens per second)
llama_print_timings: prompt eval time =     580.57 ms [/](https://file+.vscode-resource.vscode-cdn.net/)     9 tokens (   64.51 ms per token,    15.50 tokens per second)
llama_print_timings:        eval time =    6996.03 ms [/](https://file+.vscode-resource.vscode-cdn.net/)    45 runs   (  155.47 ms per token,     6.43 tokens per second)
llama_print_timings:       total time =    7822.84 ms [/](https://file+.vscode-resource.vscode-cdn.net/)    54 tokens
@ThanhNguye-n commented on GitHub (May 22, 2024): @hinthornw You have any idea why model understand this input instead of original string ? my code: ``` from langchain_core.messages import HumanMessage, AIMessage from langchain_community.llms import LlamaCpp model = LlamaCpp( temperature = 0.5, model_path = "./models/llm/Meta-Llama-3-8B-Instruct-Q5_K_M.gguf", n_ctx = 10000, n_gpu_layers = 10, n_batch = 100, # Should be between 1 and n_ctx, consider the amount of VRAM in your GPU. max_tokens = 512, verbose= True, n_threads = multiprocessing.cpu_count()-2, callback_manager = CallbackManager([StreamingStdOutCallbackHandler()]),# Callbacks support token-wise streaming streaming=True, stop=['<|eot_id|>'] ) input = [HumanMessage(content='<|begin_of_text|><|start_header_id|>user<|end_header_id|>hi, my name is Nguyen<|eot_id|><|start_header_id|>assistant<|end_header_id|>'), AIMessage(content='Nice to meet you, Nguyen! How are you doing today?'), HumanMessage(content='<|begin_of_text|><|start_header_id|>user<|end_header_id|>based on my name guess where i am from<|eot_id|><|start_header_id|>assistant<|end_header_id|>'), AIMessage(content="A fun challenge!\n\nBased on your name, Nguyen, I'm going to take a guess that you're from Vietnam. Am I correct?"), HumanMessage(content='<|begin_of_text|><|start_header_id|>user<|end_header_id|>yes, you right?<|eot_id|><|start_header_id|>assistant<|end_header_id|>')] model.invoke(input) ``` output: ``` Llama.generate: prefix-match hit I'm glad I was able to make an accurate guess! Yes, the surname "Nguyen" is very common in Vietnam. Now that we've established your Vietnamese heritage, would you like to chat about something else? llama_print_timings: load time = 1014.56 ms llama_print_timings: sample time = 27.03 ms [/](https://file+.vscode-resource.vscode-cdn.net/) 46 runs ( 0.59 ms per token, 1701.62 tokens per second) llama_print_timings: prompt eval time = 580.57 ms [/](https://file+.vscode-resource.vscode-cdn.net/) 9 tokens ( 64.51 ms per token, 15.50 tokens per second) llama_print_timings: eval time = 6996.03 ms [/](https://file+.vscode-resource.vscode-cdn.net/) 45 runs ( 155.47 ms per token, 6.43 tokens per second) llama_print_timings: total time = 7822.84 ms [/](https://file+.vscode-resource.vscode-cdn.net/) 54 tokens ```
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agents
awaiting author revision
bug
change
chat models
checkpointer
ci
code quality
code quality
dependencies
documentation
drawing
duplicate
enhancement
good first issue
help wanted
invalid
maintainer
meta
missing information
not maintained by langchain
open-swe-auto
open-swe-max
open-swe-max-auto
pending
performance
pull-request
Mirrored from GitHub Pull Request
 question
requires design
sdk
server
stale
studio
tests
type checking
usage question
v0.5
v1.1
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langchain-ai/langgraph#89
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?