Is there a way for using chat() function by passing more context ? #20

New Issue

2026-02-15T16:27:31-05:00

yindo commented

2026-02-15 16:27:31 -05:00

Originally created by @unexpand on GitHub (Jan 31, 2024).

What I am trying to attempt is something like langchain RetrievalQA, making chat request with some context.
I got the documents from pgvector postgres using search and I want to use prompt like "Use the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer." and pass the documents as context so mistral llm can use the context to respond.

my code currently looks like this and it works, but I am not sure if this is a valid approach.

async def chat( prompt:string, content:any):
  template = "Use the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer."
  {content}
  Question: {prompt}
  QA_CHAIN_PROMPT = PromptTemplate(input_variables=["context", "question"], template=template,)

  newTemplate = template + "\n" + "Question: " + prompt + "\n" + content 

  message = {'role': 'user', 'content': newTemplate }
  print(prompt, end='', flush=True)
  async for part in await AsyncClient().chat(model='mistral', messages=[message], stream=True):
    print(part['message']['content'], end='', flush=True)

Originally created by @unexpand on GitHub (Jan 31, 2024). What I am trying to attempt is something like langchain RetrievalQA, making chat request with some context. I got the documents from pgvector postgres using search and I want to use prompt like "Use the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer." and pass the documents as context so mistral llm can use the context to respond. my code currently looks like this and it works, but I am not sure if this is a valid approach. ``` async def chat( prompt:string, content:any): template = "Use the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer." {content} Question: {prompt} QA_CHAIN_PROMPT = PromptTemplate(input_variables=["context", "question"], template=template,) newTemplate = template + "\n" + "Question: " + prompt + "\n" + content message = {'role': 'user', 'content': newTemplate } print(prompt, end='', flush=True) async for part in await AsyncClient().chat(model='mistral', messages=[message], stream=True): print(part['message']['content'], end='', flush=True) ```

yindo closed this issue

2026-02-15 16:27:31 -05:00

yindo commented

2026-02-15 16:27:31 -05:00

@mxyng commented on GitHub (Jan 31, 2024):

chat is intended to be used with ollama's model templates with messages representing one side of a chat exchange. To better illustrate this here's an example equivalent to your code:

messages = [
  {
    "role": "system",
    "content": "Use the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer."
  },
  {
    "role": "user",
    "content": prompt,
  },
]

async for part in AsyncClient().chat(model='mistral', messages=messages, stream=True):
  print(part['message']['content'], end='', flush=True)

The input messages can be construct anyway you want however the only valid roles are system, user, and assistant.

@mxyng commented on GitHub (Jan 31, 2024): `chat` is intended to be used with ollama's model templates with messages representing one side of a chat exchange. To better illustrate this here's an example equivalent to your code: ```python messages = [ { "role": "system", "content": "Use the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer." }, { "role": "user", "content": prompt, }, ] async for part in AsyncClient().chat(model='mistral', messages=messages, stream=True): print(part['message']['content'], end='', flush=True) ``` The input `messages` can be construct anyway you want however the only valid roles are `system`, `user`, and `assistant`.

yindo referenced this issue

2026-02-15 16:30:00 -05:00

[PR #20] [MERGED] fix pull example typo #332

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: ollama/ollama-python#20