ollama/ollama-python
1
0
Fork 0
mirror of https://github.com/ollama/ollama-python.git synced 2026-07-21 09:05:23 -04:00
Code Issues 101 Packages Projects Releases 10 Wiki Activity

chat completion does not agree with documentation #243

New Issue
Closed
opened 2026-02-15 16:29:05 -05:00 by yindo · 9 comments
yindo commented 2026-02-15 16:29:05 -05:00
Owner
Copy Link
Copy Source

Originally created by @humblemat810 on GitHub (Mar 20, 2025).

for streaming chat api
In client that use ChatResponse
class ChatResponse(BaseGenerateResponse):
"""
Response returned by chat requests.
"""

message: Message
'Response message.'
message is required

however in documentation

https://github.com/ollama/ollama/blob/main/docs/api.md

example final response

{
"model": "llama3.2",
"created_at": "2023-08-04T19:22:45.499127Z",
"done": true,
"total_duration": 4883583458,
"load_duration": 1334875,
"prompt_eval_count": 26,
"prompt_eval_duration": 342546000,
"eval_count": 282,
"eval_duration": 4535599000
}

does not have message at all

Resolution: either:

change documentation to have message
create pydanic FinalResponse class that hsa done = True and no message required. When streaming, pydantic checking that once validation error is found, check if it matches FinalResponse type, OK if match, else raise validation error.

Originally created by @humblemat810 on GitHub (Mar 20, 2025). for streaming chat api In client that use ChatResponse class ChatResponse(BaseGenerateResponse): """ Response returned by chat requests. """ message: Message 'Response message.' message is required however in documentation https://github.com/ollama/ollama/blob/main/docs/api.md example final response { "model": "llama3.2", "created_at": "2023-08-04T19:22:45.499127Z", "done": true, "total_duration": 4883583458, "load_duration": 1334875, "prompt_eval_count": 26, "prompt_eval_duration": 342546000, "eval_count": 282, "eval_duration": 4535599000 } does not have message at all Resolution: either: change documentation to have message create pydanic FinalResponse class that hsa done = True and no message required. When streaming, pydantic checking that once validation error is found, check if it matches FinalResponse type, OK if match, else raise validation error.
yindo closed this issue 2026-02-15 16:29:06 -05:00
yindo commented 2026-02-15 16:29:06 -05:00
Author
Owner
Copy Link
Copy Source

@Mohamed0Hegazi commented on GitHub (Mar 20, 2025):

ايا

في الخميس، ٢٠ مارس ٢٠٢٥، ٩:٠٥ ص PChan @.***> كتب:

for streaming chat api
In client that use ChatResponse
class ChatResponse(BaseGenerateResponse):
"""
Response returned by chat requests.
"""

message: Message
'Response message.'
message is required

however in documentation

https://github.com/ollama/ollama/blob/main/docs/api.md

example final response

{
"model": "llama3.2",
"created_at": "2023-08-04T19:22:45.499127Z",
"done": true,
"total_duration": 4883583458,
"load_duration": 1334875,
"prompt_eval_count": 26,
"prompt_eval_duration": 342546000,
"eval_count": 282,
"eval_duration": 4535599000
}

does not have message at all

Resolution: either:

change documentation to have message
create pydanic FinalResponse class that hsa done = True and no message
required. When streaming, pydantic checking that once validation error is
found, check if it matches FinalResponse type, OK if match, else raise
validation error.

—
Reply to this email directly, view it on GitHub
https://github.com/ollama/ollama-python/issues/478, or unsubscribe
https://github.com/notifications/unsubscribe-auth/BQOY5MDAYK7WMN7GUMWSFTL2VJSCTAVCNFSM6AAAAABZMU57RKVHI2DSMVQWIX3LMV43ASLTON2WKOZSHEZTIMZSGUZTQNA
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>
[image: humblemat810]humblemat810 created an issue
(ollama/ollama-python#478)
https://github.com/ollama/ollama-python/issues/478

for streaming chat api
In client that use ChatResponse
class ChatResponse(BaseGenerateResponse):
"""
Response returned by chat requests.
"""

message: Message
'Response message.'
message is required

however in documentation

https://github.com/ollama/ollama/blob/main/docs/api.md

example final response

{
"model": "llama3.2",
"created_at": "2023-08-04T19:22:45.499127Z",
"done": true,
"total_duration": 4883583458,
"load_duration": 1334875,
"prompt_eval_count": 26,
"prompt_eval_duration": 342546000,
"eval_count": 282,
"eval_duration": 4535599000
}

does not have message at all

Resolution: either:

change documentation to have message
create pydanic FinalResponse class that hsa done = True and no message
required. When streaming, pydantic checking that once validation error is
found, check if it matches FinalResponse type, OK if match, else raise
validation error.

—
Reply to this email directly, view it on GitHub
https://github.com/ollama/ollama-python/issues/478, or unsubscribe
https://github.com/notifications/unsubscribe-auth/BQOY5MDAYK7WMN7GUMWSFTL2VJSCTAVCNFSM6AAAAABZMU57RKVHI2DSMVQWIX3LMV43ASLTON2WKOZSHEZTIMZSGUZTQNA
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

@Mohamed0Hegazi commented on GitHub (Mar 20, 2025): ايا في الخميس، ٢٠ مارس ٢٠٢٥، ٩:٠٥ ص PChan ***@***.***> كتب: > for streaming chat api > In client that use ChatResponse > class ChatResponse(BaseGenerateResponse): > """ > Response returned by chat requests. > """ > > message: Message > 'Response message.' > message is required > > however in documentation > > https://github.com/ollama/ollama/blob/main/docs/api.md > > example final response > > { > "model": "llama3.2", > "created_at": "2023-08-04T19:22:45.499127Z", > "done": true, > "total_duration": 4883583458, > "load_duration": 1334875, > "prompt_eval_count": 26, > "prompt_eval_duration": 342546000, > "eval_count": 282, > "eval_duration": 4535599000 > } > > does not have message at all > > Resolution: either: > > change documentation to have message > create pydanic FinalResponse class that hsa done = True and no message > required. When streaming, pydantic checking that once validation error is > found, check if it matches FinalResponse type, OK if match, else raise > validation error. > > — > Reply to this email directly, view it on GitHub > <https://github.com/ollama/ollama-python/issues/478>, or unsubscribe > <https://github.com/notifications/unsubscribe-auth/BQOY5MDAYK7WMN7GUMWSFTL2VJSCTAVCNFSM6AAAAABZMU57RKVHI2DSMVQWIX3LMV43ASLTON2WKOZSHEZTIMZSGUZTQNA> > . > You are receiving this because you are subscribed to this thread.Message > ID: ***@***.***> > [image: humblemat810]*humblemat810* created an issue > (ollama/ollama-python#478) > <https://github.com/ollama/ollama-python/issues/478> > > for streaming chat api > In client that use ChatResponse > class ChatResponse(BaseGenerateResponse): > """ > Response returned by chat requests. > """ > > message: Message > 'Response message.' > message is required > > however in documentation > > https://github.com/ollama/ollama/blob/main/docs/api.md > > example final response > > { > "model": "llama3.2", > "created_at": "2023-08-04T19:22:45.499127Z", > "done": true, > "total_duration": 4883583458, > "load_duration": 1334875, > "prompt_eval_count": 26, > "prompt_eval_duration": 342546000, > "eval_count": 282, > "eval_duration": 4535599000 > } > > does not have message at all > > Resolution: either: > > change documentation to have message > create pydanic FinalResponse class that hsa done = True and no message > required. When streaming, pydantic checking that once validation error is > found, check if it matches FinalResponse type, OK if match, else raise > validation error. > > — > Reply to this email directly, view it on GitHub > <https://github.com/ollama/ollama-python/issues/478>, or unsubscribe > <https://github.com/notifications/unsubscribe-auth/BQOY5MDAYK7WMN7GUMWSFTL2VJSCTAVCNFSM6AAAAABZMU57RKVHI2DSMVQWIX3LMV43ASLTON2WKOZSHEZTIMZSGUZTQNA> > . > You are receiving this because you are subscribed to this thread.Message > ID: ***@***.***> >
yindo commented 2026-02-15 16:29:06 -05:00
Author
Owner
Copy Link
Copy Source

@ParthSareen commented on GitHub (Mar 20, 2025):

@humblemat810 This is expected as the generation is streamed. The ChatResponse class also inherits from BaseGenerateResponse which gives the appropriate information

@ParthSareen commented on GitHub (Mar 20, 2025): @humblemat810 This is expected as the generation is streamed. The `ChatResponse` class also inherits from `BaseGenerateResponse` which gives the appropriate information
yindo commented 2026-02-15 16:29:07 -05:00
Author
Owner
Copy Link
Copy Source

@humblemat810 commented on GitHub (Mar 20, 2025):

However, if I stream
{
"model": "llama3.2",
"created_at": "2023-08-04T08:52:19.385406455-07:00",
"response": "The",
"done": false
}

{
"model": "llama3.2",
"created_at": "2023-08-04T19:22:45.499127Z",
"done": true,
"total_duration": 4883583458,
"load_duration": 1334875,
"prompt_eval_count": 26,
"prompt_eval_duration": 342546000,
"eval_count": 282,
"eval_duration": 4535599000
}

async def inner():
async with self._client.stream(*args, **kwargs) as r:
try:
r.raise_for_status()
except httpx.HTTPStatusError as e:
await e.response.aread()
raise ResponseError(e.response.text, e.response.status_code) from None

async for line in r.aiter_lines():
  part = json.loads(line)
  if err := part.get('error'):
    raise ResponseError(err)
  yield cls(**part)

return inner()

the line yield cls(**part)
has raised an error that complaines message is missing,
but the example does not show any missing at all

@humblemat810 commented on GitHub (Mar 20, 2025): However, if I stream { "model": "llama3.2", "created_at": "2023-08-04T08:52:19.385406455-07:00", "response": "The", "done": false } { "model": "llama3.2", "created_at": "2023-08-04T19:22:45.499127Z", "done": true, "total_duration": 4883583458, "load_duration": 1334875, "prompt_eval_count": 26, "prompt_eval_duration": 342546000, "eval_count": 282, "eval_duration": 4535599000 } async def inner(): async with self._client.stream(*args, **kwargs) as r: try: r.raise_for_status() except httpx.HTTPStatusError as e: await e.response.aread() raise ResponseError(e.response.text, e.response.status_code) from None async for line in r.aiter_lines(): part = json.loads(line) if err := part.get('error'): raise ResponseError(err) yield cls(**part) return inner() the line yield cls(**part) has raised an error that complaines message is missing, but the example does not show any missing at all
yindo commented 2026-02-15 16:29:07 -05:00
Author
Owner
Copy Link
Copy Source

@ParthSareen commented on GitHub (Mar 20, 2025):

Can you send me the exact request you're using so I can try to reproduce on my end? Thanks

@ParthSareen commented on GitHub (Mar 20, 2025): Can you send me the exact request you're using so I can try to reproduce on my end? Thanks
yindo commented 2026-02-15 16:29:07 -05:00
Author
Owner
Copy Link
Copy Source

@humblemat810 commented on GitHub (Mar 20, 2025):

I wrote a simple fastapi that just return the 2 example response

from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse
import json
import time
from datetime import datetime
import asyncio

app = FastAPI()

def current_timestamp():
    return datetime.utcnow().isoformat() + 'Z'

async def generate_responses(model: str, new_decoded_tokens: str, prompt_eval_count: int, eval_count: int):
    t_preload = time.time()
    await asyncio.sleep(1)  # Simulate processing delay
    t_prompt = time.time()
    await asyncio.sleep(1)  # Simulate processing delay
    t0 = time.time()
    await asyncio.sleep(1)  # Simulate processing delay
    t1 = time.time()

    # First JSON object
    first_response = {
        "model": "llama3.2",
        "created_at": "2023-08-04T08:52:19.385406455-07:00",
        "message": {
            "role": "assistant",
            "content": "The",
            "images": None
        },
        "done": False
        }
    yield json.dumps(first_response) + "\n"

    await asyncio.sleep(1)  # Simulate processing delay

    # Second JSON object
    final_response = {
        "model": "llama3.2",
        "created_at": "2023-08-04T19:22:45.499127Z",
        "done": True,
        "total_duration": 4883583458,
        "load_duration": 1334875,
        "prompt_eval_count": 26,
        "prompt_eval_duration": 342546000,
        "eval_count": 282,
        "eval_duration": 4535599000
        }
    yield json.dumps(final_response) + "\n"

@app.post("/api/chat")
async def chat_endpoint(request: Request):
    data = await request.json()
    model = data.get("model", "llama3.2")
    
    return StreamingResponse(
        generate_responses(model),
        media_type="application/json"
    )

===================

The error is

File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\langchain\chains\base.py", line 170, in invoke
raise e
File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\langchain\chains\base.py", line 160, in invoke
self._call(inputs, run_manager=run_manager)
File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\langchain\agents\agent.py", line 1624, in _call
next_step_output = self._take_next_step(
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\langchain\agents\agent.py", line 1330, in _take_next_step
[
File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\langchain\agents\agent.py", line 1330, in
[
File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\langchain\agents\agent.py", line 1358, in _iter_next_step
output = self._action_agent.plan(
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\langchain\agents\agent.py", line 804, in plan
full_output = self.llm_chain.predict(callbacks=callbacks, **full_inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\langchain\chains\llm.py", line 318, in predict
return self(kwargs, callbacks=callbacks)[self.output_key]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\langchain_core_api\deprecation.py", line 181, in warning_emitting_wrapper
return wrapped(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\langchain\chains\base.py", line 389, in call
return self.invoke(
^^^^^^^^^^^^
File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\langchain\chains\base.py", line 170, in invoke
raise e
File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\langchain\chains\base.py", line 160, in invoke
self._call(inputs, run_manager=run_manager)
File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\langchain\chains\llm.py", line 126, in _call
response = self.generate([inputs], run_manager=run_manager)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\langchain\chains\llm.py", line 145, in generate
results = self.llm.bind(stop=stop, **self.llm_kwargs).batch(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\langchain_core\runnables\base.py", line 5398, in batch
return self.bound.batch(
^^^^^^^^^^^^^^^^^
File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\langchain_core\runnables\base.py", line 786, in batch
return cast(list[Output], [invoke(inputs[0], configs[0])])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\langchain_core\runnables\base.py", line 782, in invoke
return self.invoke(input, config, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\langchain_core\language_models\chat_models.py", line 307, in invoke
self.generate_prompt(
File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\langchain_core\language_models\chat_models.py", line 843, in generate_prompt
return self.generate(prompt_messages, stop=stop, callbacks=callbacks, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\langchain_core\language_models\chat_models.py", line 683, in generate
self._generate_with_cache(
File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\langchain_core\language_models\chat_models.py", line 908, in _generate_with_cache
result = self._generate(
^^^^^^^^^^^^^^^
File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\langchain_ollama\chat_models.py", line 701, in _generate
final_chunk = self._chat_stream_with_aggregation(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\langchain_ollama\chat_models.py", line 602, in _chat_stream_with_aggregation
for stream_resp in self._create_chat_stream(messages, stop, **kwargs):
File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\langchain_ollama\chat_models.py", line 589, in _create_chat_stream
yield from self._client.chat(**chat_params)
File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\ollama_client.py", line 174, in inner
yield cls(**part)
^^^^^^^^^^^
File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\pydantic\main.py", line 214, in init
validated_self = self.pydantic_validator.validate_python(data, self_instance=self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 1 validation error for ChatResponse
message
Field required [type=missing, input_value={'model': 'llama3.2', 'cr...l_duration': 4535599000}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.10/v/missing

==================

the chat_params that langchain calls python_ollama (the line
line 589, in _create_chat_stream
yield from self._client.chat(**chat_params)
)is:
chat_params :

{'messages': [{'role': 'system', 'content': 'Answer the following questions as best you can. You have access to the following tools:\n\nCalculator: Useful for when you need to answer questions about math.\nwikipedia: A wrapper around Wikipedia. Useful for when you need to answer general questions about people, places, companies, facts, historical events, or other subjects. Input should be a search query.\n\nThe way you use the tools is by specifying a json blob.\nSpecifically, this json should have a `action` key (with the name of the tool to use) and a `action_input` key (with the input to the tool going here).\n\nThe only values that should be in the "action" field are: Calculator, wikipedia\n\nThe $JSON_BLOB should only contain a SINGLE action, do NOT return a list of multiple actions. Here is an example of a valid $JSON_BLOB:\n\n```\n{\n  "action": $TOOL_NAME,\n  "action_input": $INPUT\n}\n```\n\nALWAYS use the following format:\n\nQuestion: the input question you must answer\nThought: you should always think about what to do\nAction:\n```\n$JSON_BLOB\n```\nObservation: the result of the action\n... (this Thought/Action/Observation can repeat N times)\nThought: I now know the final answer\nFinal Answer: the final answer to the original input question\n\nBegin! Reminder to always use the exact characters `Final Answer` when responding.', 'images': []}, {'role': 'user', 'content': 'what is a langchain agent?\n\n', 'images': []}], 'stream': True, 'model': 'meta-llama/Llama-3.2-1B', 'format': None, 'options': Options(numa=None, num_ctx=None, num_batch=None, num_gpu=None, main_gpu=None, low_vram=None, f16_kv=None, logits_all=None, vocab_only=None, use_mmap=None, use_mlock=None, embedding_only=None, num_thread=None, num_keep=None, seed=None, num_predict=256, top_k=None, top_p=None, tfs_z=None, typical_p=None, repeat_last_n=None, temperature=0.8, repeat_penalty=None, presence_penalty=None, frequency_penalty=None, mirostat=None, mirostat_tau=None, mirostat_eta=None, penalize_newline=None, stop=['Observation:']), 'keep_alive': None, 'tools': [{'type': 'function', 'function': {'name': 'Calculator', 'description': 'Useful for when you need to answer questions about math.', 'parameters': {'properties': {'__arg1': {'title': '__arg1', 'type': 'string'}}, 'required': ['__arg1'], 'type': 'object'}}}, {'type': 'function', 'function': {'name': 'wikipedia', 'description': 'A wrapper around Wikipedia. Useful for when you need to answer general questions about people, places, companies, facts, historical events, or other subjects. Input should be a search query.', 'parameters': {'properties': {'query': {'description': 'query to look up on wikipedia', 'type': 'string'}}, 'required': ['query'], 'type': 'object'}}}]}

===================================

the environment is:

pip list
Package Version

aiohttp 3.9.3
aiosignal 1.3.1
alabaster 0.7.13
annotated-types 0.6.0
anthropic 0.49.0
anyio 4.3.0
argcomplete 3.5.1
argon2-cffi 23.1.0
argon2-cffi-bindings 21.2.0
arrow 1.3.0
asgiref 3.8.1
asttokens 2.4.1
async-lru 2.0.4
attrs 23.2.0
autodoc-pydantic 1.8.0
azure-common 1.1.28
azure-core 1.32.0
azure-search-documents 11.5.2
Babel 2.14.0
backcall 0.2.0
backoff 2.2.1
bcrypt 4.1.3
beautifulsoup4 4.12.3
black 24.3.0
bleach 6.1.0
Brotli 1.0.9
build 1.2.1
CacheControl 0.12.11
cachetools 5.3.3
certifi 2024.2.2
cffi 1.16.0
charset-normalizer 3.3.2
chroma-hnswlib 0.7.5
chromadb 0.5.4
cleo 2.1.0
click 8.1.7
codespell 2.2.6
colorama 0.4.6
coloredlogs 15.0.1
comm 0.2.2
crashtest 0.4.1
cryptography 42.0.5
dataclasses-json 0.6.4
datamodel-code-generator 0.26.3
debugpy 1.8.1
decorator 5.1.1
defusedxml 0.7.1
Deprecated 1.2.14
distlib 0.3.8
distro 1.9.0
dnspython 2.6.1
docutils 0.17.1
dulwich 0.21.3
email_validator 2.1.1
entrypoints 0.4
executing 2.0.1
fastapi 0.111.0
fastapi-cli 0.0.4
fastcore 1.4.2
fastjsonschema 2.19.1
fastrelease 0.1.17
filelock 3.13.1
flatbuffers 24.3.25
fqdn 1.5.1
frozenlist 1.4.1
fsspec 2024.6.0
genson 1.3.0
ghapi 0.1.22
google-auth 2.30.0
googleapis-common-protos 1.63.1
greenlet 3.0.3
grpcio 1.65.5
grpcio-tools 1.65.5
h11 0.14.0
h2 4.1.0
hpack 4.0.0
html5lib 1.1
httpcore 1.0.5
httptools 0.6.1
httpx 0.27.0
httpx-sse 0.4.0
huggingface-hub 0.23.4
humanfriendly 10.0
hyperframe 6.0.1
idna 3.6
imagesize 1.4.1
importlib_metadata 7.1.0
importlib_resources 6.4.0
inflect 5.6.2
iniconfig 2.0.0
installer 0.6.0
ipykernel 6.29.4
ipython 8.12.3
ipywidgets 8.1.2
isodate 0.7.2
isoduration 20.11.0
isort 5.13.2
jaraco.classes 3.2.1
jedi 0.19.1
Jinja2 3.1.3
jiter 0.5.0
json5 0.9.24
jsonpatch 1.33
jsonpointer 2.4
jsonschema 4.21.1
jsonschema-specifications 2023.12.1
jupyter 1.0.0
jupyter-cache 0.6.1
jupyter_client 7.4.9
jupyter-console 6.6.3
jupyter_core 5.7.2
jupyter-events 0.10.0
jupyter-lsp 2.2.4
jupyter_server 2.13.0
jupyter_server_terminals 0.5.3
jupyterlab 4.1.5
jupyterlab_pygments 0.3.0
jupyterlab_server 2.25.4
jupyterlab_widgets 3.0.10
keyring 23.13.1
kubernetes 30.1.0
langchain 0.3.21
langchain-anthropic 0.3.10
langchain-community 0.3.20
langchain-core 0.3.45
langchain-experimental 0.3.4
langchain-ollama 0.2.3
langchain-openai 0.3.9
langchain-text-splitters 0.3.7
langgraph 0.3.15
langgraph-checkpoint 2.0.21
langgraph-prebuilt 0.1.3
langgraph-sdk 0.1.57
langsmith 0.3.0
LinkChecker 10.3.0
livereload 2.6.3
lockfile 0.12.2
markdown-it-py 2.2.0
MarkupSafe 2.1.5
marshmallow 3.21.1
matplotlib-inline 0.1.6
mdit-py-plugins 0.3.5
mdurl 0.1.2
mem0ai 0.1.70
mistune 3.0.2
mkl-fft 1.3.8
mkl-random 1.2.4
mkl-service 2.4.0
mmh3 4.1.0
monotonic 1.6
more-itertools 10.1.0
mpmath 1.3.0
msgpack 1.1.0
multidict 6.0.5
mypy-extensions 1.0.0
myst-nb 0.17.2
myst-parser 0.18.1
nbclient 0.7.4
nbconvert 7.16.3
nbdev 1.2.0
nbdoc 0.0.82
nbformat 5.10.3
nbsphinx 0.8.12
neo4j 5.24.0
nest-asyncio 1.6.0
notebook 7.1.2
notebook_shim 0.2.4
numexpr 2.10.2
numpy 1.26.4
numpydoc 1.2
oauthlib 3.2.2
ollama 0.4.7
onnxruntime 1.18.0
openai 1.66.5
opentelemetry-api 1.25.0
opentelemetry-exporter-otlp-proto-common 1.25.0
opentelemetry-exporter-otlp-proto-grpc 1.25.0
opentelemetry-instrumentation 0.46b0
opentelemetry-instrumentation-asgi 0.46b0
opentelemetry-instrumentation-fastapi 0.46b0
opentelemetry-proto 1.25.0
opentelemetry-sdk 1.25.0
opentelemetry-semantic-conventions 0.46b0
opentelemetry-util-http 0.46b0
orjson 3.10.15
overrides 7.7.0
packaging 23.2
pandocfilters 1.5.1
parso 0.8.3
pathspec 0.12.1
pexpect 4.8.0
pickleshare 0.7.5
pip 24.0
pkginfo 1.10.0
platformdirs 4.2.0
pluggy 1.5.0
poetry 1.4.0
poetry-core 1.5.1
poetry-plugin-export 1.3.0
portalocker 2.10.1
posthog 3.5.0
prometheus_client 0.20.0
prompt-toolkit 3.0.43
protobuf 5.28.2
psutil 5.9.8
psycopg2-binary 2.9.10
ptyprocess 0.7.0
pure-eval 0.2.2
pyasn1 0.6.0
pyasn1_modules 0.4.0
pycparser 2.22
pydantic 2.10.6
pydantic_core 2.27.2
pydantic-settings 2.8.1
pydata-sphinx-theme 0.8.1
Pygments 2.17.2
pyodbc 5.2.0
pyOpenSSL 24.0.0
PyPika 0.48.9
pyproject_hooks 1.0.0
pyreadline3 3.4.1
PySocks 1.7.1
pytest 8.2.2
python-dateutil 2.9.0.post0
python-dotenv 1.0.1
python-json-logger 2.0.7
python-multipart 0.0.9
pytz 2024.1
pywin32 306
pywin32-ctypes 0.2.2
pywinpty 2.0.13
PyYAML 6.0.1
pyzmq 25.1.2
qdrant-client 1.11.0
qtconsole 5.5.1
QtPy 2.4.1
rank-bm25 0.2.2
rapidfuzz 3.5.2
referencing 0.34.0
regex 2023.12.25
requests 2.31.0
requests-oauthlib 2.0.0
requests-toolbelt 1.0.0
rfc3339-validator 0.1.4
rfc3986-validator 0.1.1
rich 13.7.1
rpds-py 0.18.0
rsa 4.9
ruff 0.4.2
Send2Trash 1.8.2
setuptools 69.5.1
shellingham 1.5.0
six 1.16.0
sniffio 1.3.1
snowballstemmer 2.2.0
soupsieve 2.5
Sphinx 4.5.0
sphinx-autobuild 2021.3.14
sphinx-book-theme 0.3.3
sphinx-copybutton 0.5.2
sphinx-panels 0.6.0
sphinx-rtd-theme 1.3.0
sphinx-typlog-theme 0.8.0
sphinxcontrib-applehelp 1.0.4
sphinxcontrib-devhelp 1.0.2
sphinxcontrib-htmlhelp 2.0.1
sphinxcontrib-jquery 4.1
sphinxcontrib-jsmath 1.0.1
sphinxcontrib-qthelp 1.0.3
sphinxcontrib-serializinghtml 1.1.5
SQLAlchemy 2.0.32
stack-data 0.6.3
starlette 0.37.2
sympy 1.12.1
tabulate 0.9.0
tenacity 8.2.3
terminado 0.18.1
tiktoken 0.7.0
tinycss2 1.2.1
tokenize-rt 5.2.0
tokenizers 0.19.1
toml 0.10.2
tomlkit 0.11.1
tornado 6.4
tqdm 4.66.2
traitlets 5.14.2
trove-classifiers 2023.10.18
typer 0.12.3
types-python-dateutil 2.9.0.20240316
typing_extensions 4.12.2
typing-inspect 0.9.0
ujson 5.10.0
uri-template 1.3.0
urllib3 2.2.1
uvicorn 0.30.1
virtualenv 20.16.2
watchfiles 0.22.0
wcwidth 0.2.13
webcolors 1.13
webencodings 0.5.1
websocket-client 1.7.0
websockets 12.0
wheel 0.43.0
widgetsnbextension 4.0.10
wikipedia 1.4.0
win-inet-pton 1.1.0
wrapt 1.16.0
yarl 1.9.4
zipp 3.18.1
zstandard 0.23.0

@humblemat810 commented on GitHub (Mar 20, 2025): I wrote a simple fastapi that just return the 2 example response ``` python from fastapi import FastAPI, Request from fastapi.responses import StreamingResponse import json import time from datetime import datetime import asyncio app = FastAPI() def current_timestamp(): return datetime.utcnow().isoformat() + 'Z' async def generate_responses(model: str, new_decoded_tokens: str, prompt_eval_count: int, eval_count: int): t_preload = time.time() await asyncio.sleep(1) # Simulate processing delay t_prompt = time.time() await asyncio.sleep(1) # Simulate processing delay t0 = time.time() await asyncio.sleep(1) # Simulate processing delay t1 = time.time() # First JSON object first_response = { "model": "llama3.2", "created_at": "2023-08-04T08:52:19.385406455-07:00", "message": { "role": "assistant", "content": "The", "images": None }, "done": False } yield json.dumps(first_response) + "\n" await asyncio.sleep(1) # Simulate processing delay # Second JSON object final_response = { "model": "llama3.2", "created_at": "2023-08-04T19:22:45.499127Z", "done": True, "total_duration": 4883583458, "load_duration": 1334875, "prompt_eval_count": 26, "prompt_eval_duration": 342546000, "eval_count": 282, "eval_duration": 4535599000 } yield json.dumps(final_response) + "\n" @app.post("/api/chat") async def chat_endpoint(request: Request): data = await request.json() model = data.get("model", "llama3.2") return StreamingResponse( generate_responses(model), media_type="application/json" ) ``` =================== The error is File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\langchain\chains\base.py", line 170, in invoke raise e File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\langchain\chains\base.py", line 160, in invoke self._call(inputs, run_manager=run_manager) File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\langchain\agents\agent.py", line 1624, in _call next_step_output = self._take_next_step( ^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\langchain\agents\agent.py", line 1330, in _take_next_step [ File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\langchain\agents\agent.py", line 1330, in <listcomp> [ File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\langchain\agents\agent.py", line 1358, in _iter_next_step output = self._action_agent.plan( ^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\langchain\agents\agent.py", line 804, in plan full_output = self.llm_chain.predict(callbacks=callbacks, **full_inputs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\langchain\chains\llm.py", line 318, in predict return self(kwargs, callbacks=callbacks)[self.output_key] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\langchain_core\_api\deprecation.py", line 181, in warning_emitting_wrapper return wrapped(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\langchain\chains\base.py", line 389, in __call__ return self.invoke( ^^^^^^^^^^^^ File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\langchain\chains\base.py", line 170, in invoke raise e File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\langchain\chains\base.py", line 160, in invoke self._call(inputs, run_manager=run_manager) File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\langchain\chains\llm.py", line 126, in _call response = self.generate([inputs], run_manager=run_manager) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\langchain\chains\llm.py", line 145, in generate results = self.llm.bind(stop=stop, **self.llm_kwargs).batch( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\langchain_core\runnables\base.py", line 5398, in batch return self.bound.batch( ^^^^^^^^^^^^^^^^^ File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\langchain_core\runnables\base.py", line 786, in batch return cast(list[Output], [invoke(inputs[0], configs[0])]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\langchain_core\runnables\base.py", line 782, in invoke return self.invoke(input, config, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\langchain_core\language_models\chat_models.py", line 307, in invoke self.generate_prompt( File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\langchain_core\language_models\chat_models.py", line 843, in generate_prompt return self.generate(prompt_messages, stop=stop, callbacks=callbacks, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\langchain_core\language_models\chat_models.py", line 683, in generate self._generate_with_cache( File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\langchain_core\language_models\chat_models.py", line 908, in _generate_with_cache result = self._generate( ^^^^^^^^^^^^^^^ File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\langchain_ollama\chat_models.py", line 701, in _generate final_chunk = self._chat_stream_with_aggregation( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\langchain_ollama\chat_models.py", line 602, in _chat_stream_with_aggregation for stream_resp in self._create_chat_stream(messages, stop, **kwargs): File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\langchain_ollama\chat_models.py", line 589, in _create_chat_stream yield from self._client.chat(**chat_params) File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\ollama\_client.py", line 174, in inner yield cls(**part) ^^^^^^^^^^^ File "C:\Users\humblemat\anaconda3\envs\langchain_dev\Lib\site-packages\pydantic\main.py", line 214, in __init__ validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pydantic_core._pydantic_core.ValidationError: 1 validation error for ChatResponse message Field required [type=missing, input_value={'model': 'llama3.2', 'cr...l_duration': 4535599000}, input_type=dict] For further information visit https://errors.pydantic.dev/2.10/v/missing ================== the chat_params that langchain calls python_ollama (the line line 589, in _create_chat_stream yield from self._client.chat(**chat_params) )is: chat_params : ``` {'messages': [{'role': 'system', 'content': 'Answer the following questions as best you can. You have access to the following tools:\n\nCalculator: Useful for when you need to answer questions about math.\nwikipedia: A wrapper around Wikipedia. Useful for when you need to answer general questions about people, places, companies, facts, historical events, or other subjects. Input should be a search query.\n\nThe way you use the tools is by specifying a json blob.\nSpecifically, this json should have a `action` key (with the name of the tool to use) and a `action_input` key (with the input to the tool going here).\n\nThe only values that should be in the "action" field are: Calculator, wikipedia\n\nThe $JSON_BLOB should only contain a SINGLE action, do NOT return a list of multiple actions. Here is an example of a valid $JSON_BLOB:\n\n```\n{\n "action": $TOOL_NAME,\n "action_input": $INPUT\n}\n```\n\nALWAYS use the following format:\n\nQuestion: the input question you must answer\nThought: you should always think about what to do\nAction:\n```\n$JSON_BLOB\n```\nObservation: the result of the action\n... (this Thought/Action/Observation can repeat N times)\nThought: I now know the final answer\nFinal Answer: the final answer to the original input question\n\nBegin! Reminder to always use the exact characters `Final Answer` when responding.', 'images': []}, {'role': 'user', 'content': 'what is a langchain agent?\n\n', 'images': []}], 'stream': True, 'model': 'meta-llama/Llama-3.2-1B', 'format': None, 'options': Options(numa=None, num_ctx=None, num_batch=None, num_gpu=None, main_gpu=None, low_vram=None, f16_kv=None, logits_all=None, vocab_only=None, use_mmap=None, use_mlock=None, embedding_only=None, num_thread=None, num_keep=None, seed=None, num_predict=256, top_k=None, top_p=None, tfs_z=None, typical_p=None, repeat_last_n=None, temperature=0.8, repeat_penalty=None, presence_penalty=None, frequency_penalty=None, mirostat=None, mirostat_tau=None, mirostat_eta=None, penalize_newline=None, stop=['Observation:']), 'keep_alive': None, 'tools': [{'type': 'function', 'function': {'name': 'Calculator', 'description': 'Useful for when you need to answer questions about math.', 'parameters': {'properties': {'__arg1': {'title': '__arg1', 'type': 'string'}}, 'required': ['__arg1'], 'type': 'object'}}}, {'type': 'function', 'function': {'name': 'wikipedia', 'description': 'A wrapper around Wikipedia. Useful for when you need to answer general questions about people, places, companies, facts, historical events, or other subjects. Input should be a search query.', 'parameters': {'properties': {'query': {'description': 'query to look up on wikipedia', 'type': 'string'}}, 'required': ['query'], 'type': 'object'}}}]} ``` =================================== the environment is: ===================================== pip list Package Version ---------------------------------------- -------------- aiohttp 3.9.3 aiosignal 1.3.1 alabaster 0.7.13 annotated-types 0.6.0 anthropic 0.49.0 anyio 4.3.0 argcomplete 3.5.1 argon2-cffi 23.1.0 argon2-cffi-bindings 21.2.0 arrow 1.3.0 asgiref 3.8.1 asttokens 2.4.1 async-lru 2.0.4 attrs 23.2.0 autodoc-pydantic 1.8.0 azure-common 1.1.28 azure-core 1.32.0 azure-search-documents 11.5.2 Babel 2.14.0 backcall 0.2.0 backoff 2.2.1 bcrypt 4.1.3 beautifulsoup4 4.12.3 black 24.3.0 bleach 6.1.0 Brotli 1.0.9 build 1.2.1 CacheControl 0.12.11 cachetools 5.3.3 certifi 2024.2.2 cffi 1.16.0 charset-normalizer 3.3.2 chroma-hnswlib 0.7.5 chromadb 0.5.4 cleo 2.1.0 click 8.1.7 codespell 2.2.6 colorama 0.4.6 coloredlogs 15.0.1 comm 0.2.2 crashtest 0.4.1 cryptography 42.0.5 dataclasses-json 0.6.4 datamodel-code-generator 0.26.3 debugpy 1.8.1 decorator 5.1.1 defusedxml 0.7.1 Deprecated 1.2.14 distlib 0.3.8 distro 1.9.0 dnspython 2.6.1 docutils 0.17.1 dulwich 0.21.3 email_validator 2.1.1 entrypoints 0.4 executing 2.0.1 fastapi 0.111.0 fastapi-cli 0.0.4 fastcore 1.4.2 fastjsonschema 2.19.1 fastrelease 0.1.17 filelock 3.13.1 flatbuffers 24.3.25 fqdn 1.5.1 frozenlist 1.4.1 fsspec 2024.6.0 genson 1.3.0 ghapi 0.1.22 google-auth 2.30.0 googleapis-common-protos 1.63.1 greenlet 3.0.3 grpcio 1.65.5 grpcio-tools 1.65.5 h11 0.14.0 h2 4.1.0 hpack 4.0.0 html5lib 1.1 httpcore 1.0.5 httptools 0.6.1 httpx 0.27.0 httpx-sse 0.4.0 huggingface-hub 0.23.4 humanfriendly 10.0 hyperframe 6.0.1 idna 3.6 imagesize 1.4.1 importlib_metadata 7.1.0 importlib_resources 6.4.0 inflect 5.6.2 iniconfig 2.0.0 installer 0.6.0 ipykernel 6.29.4 ipython 8.12.3 ipywidgets 8.1.2 isodate 0.7.2 isoduration 20.11.0 isort 5.13.2 jaraco.classes 3.2.1 jedi 0.19.1 Jinja2 3.1.3 jiter 0.5.0 json5 0.9.24 jsonpatch 1.33 jsonpointer 2.4 jsonschema 4.21.1 jsonschema-specifications 2023.12.1 jupyter 1.0.0 jupyter-cache 0.6.1 jupyter_client 7.4.9 jupyter-console 6.6.3 jupyter_core 5.7.2 jupyter-events 0.10.0 jupyter-lsp 2.2.4 jupyter_server 2.13.0 jupyter_server_terminals 0.5.3 jupyterlab 4.1.5 jupyterlab_pygments 0.3.0 jupyterlab_server 2.25.4 jupyterlab_widgets 3.0.10 keyring 23.13.1 kubernetes 30.1.0 langchain 0.3.21 langchain-anthropic 0.3.10 langchain-community 0.3.20 langchain-core 0.3.45 langchain-experimental 0.3.4 langchain-ollama 0.2.3 langchain-openai 0.3.9 langchain-text-splitters 0.3.7 langgraph 0.3.15 langgraph-checkpoint 2.0.21 langgraph-prebuilt 0.1.3 langgraph-sdk 0.1.57 langsmith 0.3.0 LinkChecker 10.3.0 livereload 2.6.3 lockfile 0.12.2 markdown-it-py 2.2.0 MarkupSafe 2.1.5 marshmallow 3.21.1 matplotlib-inline 0.1.6 mdit-py-plugins 0.3.5 mdurl 0.1.2 mem0ai 0.1.70 mistune 3.0.2 mkl-fft 1.3.8 mkl-random 1.2.4 mkl-service 2.4.0 mmh3 4.1.0 monotonic 1.6 more-itertools 10.1.0 mpmath 1.3.0 msgpack 1.1.0 multidict 6.0.5 mypy-extensions 1.0.0 myst-nb 0.17.2 myst-parser 0.18.1 nbclient 0.7.4 nbconvert 7.16.3 nbdev 1.2.0 nbdoc 0.0.82 nbformat 5.10.3 nbsphinx 0.8.12 neo4j 5.24.0 nest-asyncio 1.6.0 notebook 7.1.2 notebook_shim 0.2.4 numexpr 2.10.2 numpy 1.26.4 numpydoc 1.2 oauthlib 3.2.2 ollama 0.4.7 onnxruntime 1.18.0 openai 1.66.5 opentelemetry-api 1.25.0 opentelemetry-exporter-otlp-proto-common 1.25.0 opentelemetry-exporter-otlp-proto-grpc 1.25.0 opentelemetry-instrumentation 0.46b0 opentelemetry-instrumentation-asgi 0.46b0 opentelemetry-instrumentation-fastapi 0.46b0 opentelemetry-proto 1.25.0 opentelemetry-sdk 1.25.0 opentelemetry-semantic-conventions 0.46b0 opentelemetry-util-http 0.46b0 orjson 3.10.15 overrides 7.7.0 packaging 23.2 pandocfilters 1.5.1 parso 0.8.3 pathspec 0.12.1 pexpect 4.8.0 pickleshare 0.7.5 pip 24.0 pkginfo 1.10.0 platformdirs 4.2.0 pluggy 1.5.0 poetry 1.4.0 poetry-core 1.5.1 poetry-plugin-export 1.3.0 portalocker 2.10.1 posthog 3.5.0 prometheus_client 0.20.0 prompt-toolkit 3.0.43 protobuf 5.28.2 psutil 5.9.8 psycopg2-binary 2.9.10 ptyprocess 0.7.0 pure-eval 0.2.2 pyasn1 0.6.0 pyasn1_modules 0.4.0 pycparser 2.22 pydantic 2.10.6 pydantic_core 2.27.2 pydantic-settings 2.8.1 pydata-sphinx-theme 0.8.1 Pygments 2.17.2 pyodbc 5.2.0 pyOpenSSL 24.0.0 PyPika 0.48.9 pyproject_hooks 1.0.0 pyreadline3 3.4.1 PySocks 1.7.1 pytest 8.2.2 python-dateutil 2.9.0.post0 python-dotenv 1.0.1 python-json-logger 2.0.7 python-multipart 0.0.9 pytz 2024.1 pywin32 306 pywin32-ctypes 0.2.2 pywinpty 2.0.13 PyYAML 6.0.1 pyzmq 25.1.2 qdrant-client 1.11.0 qtconsole 5.5.1 QtPy 2.4.1 rank-bm25 0.2.2 rapidfuzz 3.5.2 referencing 0.34.0 regex 2023.12.25 requests 2.31.0 requests-oauthlib 2.0.0 requests-toolbelt 1.0.0 rfc3339-validator 0.1.4 rfc3986-validator 0.1.1 rich 13.7.1 rpds-py 0.18.0 rsa 4.9 ruff 0.4.2 Send2Trash 1.8.2 setuptools 69.5.1 shellingham 1.5.0 six 1.16.0 sniffio 1.3.1 snowballstemmer 2.2.0 soupsieve 2.5 Sphinx 4.5.0 sphinx-autobuild 2021.3.14 sphinx-book-theme 0.3.3 sphinx-copybutton 0.5.2 sphinx-panels 0.6.0 sphinx-rtd-theme 1.3.0 sphinx-typlog-theme 0.8.0 sphinxcontrib-applehelp 1.0.4 sphinxcontrib-devhelp 1.0.2 sphinxcontrib-htmlhelp 2.0.1 sphinxcontrib-jquery 4.1 sphinxcontrib-jsmath 1.0.1 sphinxcontrib-qthelp 1.0.3 sphinxcontrib-serializinghtml 1.1.5 SQLAlchemy 2.0.32 stack-data 0.6.3 starlette 0.37.2 sympy 1.12.1 tabulate 0.9.0 tenacity 8.2.3 terminado 0.18.1 tiktoken 0.7.0 tinycss2 1.2.1 tokenize-rt 5.2.0 tokenizers 0.19.1 toml 0.10.2 tomlkit 0.11.1 tornado 6.4 tqdm 4.66.2 traitlets 5.14.2 trove-classifiers 2023.10.18 typer 0.12.3 types-python-dateutil 2.9.0.20240316 typing_extensions 4.12.2 typing-inspect 0.9.0 ujson 5.10.0 uri-template 1.3.0 urllib3 2.2.1 uvicorn 0.30.1 virtualenv 20.16.2 watchfiles 0.22.0 wcwidth 0.2.13 webcolors 1.13 webencodings 0.5.1 websocket-client 1.7.0 websockets 12.0 wheel 0.43.0 widgetsnbextension 4.0.10 wikipedia 1.4.0 win-inet-pton 1.1.0 wrapt 1.16.0 yarl 1.9.4 zipp 3.18.1 zstandard 0.23.0
yindo commented 2026-02-15 16:29:08 -05:00
Author
Owner
Copy Link
Copy Source

@humblemat810 commented on GitHub (Mar 20, 2025):

this above sample debug server just stream the 2 example responses in the documentation MD file without checking or running any real LLMs, serving as a minimal example.
and the below is the client code (wrapped in pytest test case):

    from langchain.agents import load_tools, initialize_agent, tool
    from langchain.agents import AgentType
    
    tools = load_tools(["llm-math", "wikipedia"], llm=OllamaLlama3dot1)
    llm_with_tools = OllamaLlama3dot1.bind_tools(tools)
    agent = initialize_agent(
        tools = tools,
        llm = llm_with_tools,
        agent=AgentType.CHAT_ZERO_SHOT_REACT_DESCRIPTION,
        handle_parsing_errors=True,
        verbose=True)
    while True:
        try:
            response = agent.invoke("what is a langchain agent?")
            print(response)
            break
        except BaseException as e:
            breakpoint()
            pass
        
        pass
@humblemat810 commented on GitHub (Mar 20, 2025): this above sample debug server just stream the 2 example responses in the documentation MD file without checking or running any real LLMs, serving as a minimal example. and the below is the client code (wrapped in pytest test case): ``` from langchain.agents import load_tools, initialize_agent, tool from langchain.agents import AgentType tools = load_tools(["llm-math", "wikipedia"], llm=OllamaLlama3dot1) llm_with_tools = OllamaLlama3dot1.bind_tools(tools) agent = initialize_agent( tools = tools, llm = llm_with_tools, agent=AgentType.CHAT_ZERO_SHOT_REACT_DESCRIPTION, handle_parsing_errors=True, verbose=True) while True: try: response = agent.invoke("what is a langchain agent?") print(response) break except BaseException as e: breakpoint() pass pass ```
yindo commented 2026-02-15 16:29:08 -05:00
Author
Owner
Copy Link
Copy Source

@ParthSareen commented on GitHub (Mar 20, 2025):

Ah I see the issue. Yes the message should still be included as part of the final response. Will update this - thanks for bringing up!

@ParthSareen commented on GitHub (Mar 20, 2025): Ah I see the issue. Yes the message should still be included as part of the final response. Will update this - thanks for bringing up!
yindo commented 2026-02-15 16:29:08 -05:00
Author
Owner
Copy Link
Copy Source

@humblemat810 commented on GitHub (Mar 20, 2025):

The problem is, is there any central specification/requirements for ollama and/or python_ollama?

Here is also a closely relate issue near the point the previous error is raised.

Situation:
I am making an Ollama API compliant intermediate and found this. since it use aiter_lines.

Issue:
And it seems to require each packet to be delimited by a new line character. Which is also not documented. If a series of json is sent without newline, there is a json parsing error. So either change the algorithm to accept jsons with or without new line, or the documentation need to specify the series of jsons has to be delimited by newlines.

Discussion:
is it decided that each streaming chunk has to be interleaved by newline? series of Json chunks can be parsed with or without newline in between. without newline, we need to keep track of the open close brackets to, at the expense of transferring an extra new line. While with newlines, just aiter_lines can easily parse the json streams at the expense of extra newline character.

@humblemat810 commented on GitHub (Mar 20, 2025): The problem is, is there any central specification/requirements for ollama and/or python_ollama? Here is also a closely relate issue near the point the previous error is raised. Situation: I am making an Ollama API compliant intermediate and found this. since it use aiter_lines. Issue: And it seems to require each packet to be delimited by a new line character. Which is also not documented. If a series of json is sent without newline, there is a json parsing error. So either change the algorithm to accept jsons with or without new line, or the documentation need to specify the series of jsons has to be delimited by newlines. Discussion: is it decided that each streaming chunk has to be interleaved by newline? series of Json chunks can be parsed with or without newline in between. without newline, we need to keep track of the open close brackets to, at the expense of transferring an extra new line. While with newlines, just aiter_lines can easily parse the json streams at the expense of extra newline character.
yindo commented 2026-02-15 16:29:08 -05:00
Author
Owner
Copy Link
Copy Source

@Mohamed0Hegazi commented on GitHub (Mar 20, 2025):

Thanks for your explanation.

في الجمعة، ٢١ مارس ٢٠٢٥، ٢:٤٩ ص PChan @.***> كتب:

The problem is, is there any central specification/requirements for ollama
and/or python_ollama?

I am making an Ollama API compliant intermediate and found this. since it
use aiter_lines.

And it seems to require each packet to be delimited by a new line
character. Which is also not documented.
is it decided that each streaming chunk has to be interleaved by newline?
series of Json chunks can be parsed with or without newline in between.
without newline, we need to keep track of the open close brackets to, at
the expense of transferring an extra new line. While with newlines, just
aiter_lines can easily parse the json streams at the expense of extra
newline character.

—
Reply to this email directly, view it on GitHub
https://github.com/ollama/ollama-python/issues/478#issuecomment-2741968441,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/BQOY5MHBFM2KDDW7P3NEBE32VNOYZAVCNFSM6AAAAABZMU57RKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDONBRHE3DQNBUGE
.
You are receiving this because you commented.Message ID:
@.***>
[image: humblemat810]humblemat810 left a comment
(ollama/ollama-python#478)
https://github.com/ollama/ollama-python/issues/478#issuecomment-2741968441

The problem is, is there any central specification/requirements for ollama
and/or python_ollama?

I am making an Ollama API compliant intermediate and found this. since it
use aiter_lines.

And it seems to require each packet to be delimited by a new line
character. Which is also not documented.
is it decided that each streaming chunk has to be interleaved by newline?
series of Json chunks can be parsed with or without newline in between.
without newline, we need to keep track of the open close brackets to, at
the expense of transferring an extra new line. While with newlines, just
aiter_lines can easily parse the json streams at the expense of extra
newline character.

—
Reply to this email directly, view it on GitHub
https://github.com/ollama/ollama-python/issues/478#issuecomment-2741968441,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/BQOY5MHBFM2KDDW7P3NEBE32VNOYZAVCNFSM6AAAAABZMU57RKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDONBRHE3DQNBUGE
.
You are receiving this because you commented.Message ID:
@.***>

@Mohamed0Hegazi commented on GitHub (Mar 20, 2025): Thanks for your explanation. في الجمعة، ٢١ مارس ٢٠٢٥، ٢:٤٩ ص PChan ***@***.***> كتب: > The problem is, is there any central specification/requirements for ollama > and/or python_ollama? > > I am making an Ollama API compliant intermediate and found this. since it > use aiter_lines. > > And it seems to require each packet to be delimited by a new line > character. Which is also not documented. > is it decided that each streaming chunk has to be interleaved by newline? > series of Json chunks can be parsed with or without newline in between. > without newline, we need to keep track of the open close brackets to, at > the expense of transferring an extra new line. While with newlines, just > aiter_lines can easily parse the json streams at the expense of extra > newline character. > > — > Reply to this email directly, view it on GitHub > <https://github.com/ollama/ollama-python/issues/478#issuecomment-2741968441>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/BQOY5MHBFM2KDDW7P3NEBE32VNOYZAVCNFSM6AAAAABZMU57RKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDONBRHE3DQNBUGE> > . > You are receiving this because you commented.Message ID: > ***@***.***> > [image: humblemat810]*humblemat810* left a comment > (ollama/ollama-python#478) > <https://github.com/ollama/ollama-python/issues/478#issuecomment-2741968441> > > The problem is, is there any central specification/requirements for ollama > and/or python_ollama? > > I am making an Ollama API compliant intermediate and found this. since it > use aiter_lines. > > And it seems to require each packet to be delimited by a new line > character. Which is also not documented. > is it decided that each streaming chunk has to be interleaved by newline? > series of Json chunks can be parsed with or without newline in between. > without newline, we need to keep track of the open close brackets to, at > the expense of transferring an extra new line. While with newlines, just > aiter_lines can easily parse the json streams at the expense of extra > newline character. > > — > Reply to this email directly, view it on GitHub > <https://github.com/ollama/ollama-python/issues/478#issuecomment-2741968441>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/BQOY5MHBFM2KDDW7P3NEBE32VNOYZAVCNFSM6AAAAABZMU57RKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDONBRHE3DQNBUGE> > . > You are receiving this because you commented.Message ID: > ***@***.***> >
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
enhancement
pull-request
Mirrored from GitHub Pull Request
 thinking
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: ollama/ollama-python#243
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?