langchain-ai/langserve
1
0
Fork 0
mirror of https://github.com/langchain-ai/langserve.git synced 2026-07-16 09:34:30 -04:00
Code Issues 136 Packages Projects Releases 10 Wiki Activity

Bad character in state causes incomplete streaming results, leading RemoteRunnable to issues #252

New Issue
Closed
opened 2026-02-16 00:19:49 -05:00 by yindo · 3 comments
yindo commented 2026-02-16 00:19:49 -05:00
Owner
Copy Link
Copy Source

Originally created by @fred913 on GitHub (Sep 9, 2025).

How do we find it

Recently we got the following error in our product using RemoteRunnable. (reproduced error in a minimal client, so error logs is a bit different, but the core is same)

Just to note that, using root in a production environment is a bad idea, so don't imitate us

(venv) (base) root@--our-server--:~/tools/searchTools_langgraph# python client.py stream
SearchTools SubGraph Client Test Suite
==================================================

=== Testing Streaming Invocation ===
Searching for: XXXXX
2025-09-10 09:36:29.676 | ERROR    | __main__:test_stream:152 - Error during streaming: unexpected end of data: line 1 column 6604 (char 6603)
Traceback (most recent call last):

  File "{PROJECT}/client.py", line 266, in <module>
    success = test_stream()
              └ <function test_stream at 0x7f765977b7e0>

> File "{PROJECT}/client.py", line 117, in test_stream
    for chunk in subgraph.stream(
                 │        └ <function RemoteRunnable.stream at 0x7f765673ab60>
                 └ <langserve.client.RemoteRunnable object at 0x7f7656751400>

  File "{PROJECT}/venv/lib/python3.13/site-packages/langserve/client.py", line 551, in stream
    chunk = self._lc_serializer.loads(sse["data"])
            │    │              │     └ {'event': 'data', 'data': '{"search":{"query":"XXXXXXXXX",...
            │    │              └ <function Serializer.loads at 0x7f7656819ee0>
            │    └ <langserve.serialization.WellKnownLCSerializer object at 0x7f76565a82f0>
            └ <langserve.client.RemoteRunnable object at 0x7f7656751400>
  File "{PROJECT}/venv/lib/python3.13/site-packages/langserve/serialization.py", line 166, in loads
    return self.loadd(orjson.loads(s))
           │    │     │      │     └ '{"search":{"query":"XXXXXXX","max_results...
           │    │     │      └ <built-in function loads>
           │    │     └ <module 'orjson' from '{PROJECT}/venv/lib/python3.13/site-packages/orjson/__init__.py'>
           │    └ <function WellKnownLCSerializer.loadd at 0x7f765681a200>
           └ <langserve.serialization.WellKnownLCSerializer object at 0x7f76565a82f0>

orjson.JSONDecodeError: unexpected end of data: line 1 column 6604 (char 6603)

This is weird and we thought it makes no sense. You see, HTTP is TCP-based, so even though we don't have HTTPS set up for internal servers, partial JSON shouldn't appear in responses.

We set up coredumpy and dowhen in our environment, and we did this.

from dowhen import when


def _cb1(chunk: Any):
    logger.warning(chunk)

when(langserve.APIHandler.stream, "yield {").do(_cb1)
when(langserve.APIHandler.stream, "yield {").do('__import__("coredumpy").dump()')

Coredumpy is a post-mortem tool that stores the current frame state into a static dump file, so you can load and run a Pdb-like debugging console afterwards. Dowhen is a tool that injects a bit of code into a callable, so it's equivent to adding the code below the marked line (marked via the code keyword specified in when()). We did set it up for the test_stream() (in the test client) above, so we got the json data where an unexpected end of file appear. This way we got the error json happened in test_stream(), and the original state dumped as json (via orjson, same way as WellKnownLCSerializer)

Image

AFAIK, Python's embedded json.dumps has a ensure_ascii: bool that prevents those characters from a serialized JSON string, but I didn't see one in orjson. We implement one with json.dumps instead, below:

class NotWellKnownSerializer(langserve.serialization.Serializer):
    def dumps(self, obj: Any) -> bytes:
        """Dump the given object to a JSON byte string."""
        return json.dumps(obj, default=langserve.serialization.default, ensure_ascii=True).encode("utf-8")

    def loadd(self, s: Any) -> Any:
        return langserve.serialization._decode_lc_objects(s)

And we re-run our test_client, see it just works. Unfortunately I don't have screenshots for this, but it really just works fine then. The client gets the full fine response.

I'm filing this issue since I'm not sure if this issue belongs to LangServe or FastAPI, or starlette_sse, so any guidance on whether this is expected behavior, a known limitation, or something that should be fixed at the LangServe level, would be much appreciated. Thanks for maintaining this project by the way!

Originally created by @fred913 on GitHub (Sep 9, 2025). ## How do we find it Recently we got the following error in our product using RemoteRunnable. (*reproduced* error in a minimal client, so error logs is a bit different, but the core is same) *Just to note that, using root in a production environment is a bad idea, so don't imitate us* ```bash (venv) (base) root@--our-server--:~/tools/searchTools_langgraph# python client.py stream SearchTools SubGraph Client Test Suite ================================================== === Testing Streaming Invocation === Searching for: XXXXX 2025-09-10 09:36:29.676 | ERROR | __main__:test_stream:152 - Error during streaming: unexpected end of data: line 1 column 6604 (char 6603) Traceback (most recent call last): File "{PROJECT}/client.py", line 266, in <module> success = test_stream() └ <function test_stream at 0x7f765977b7e0> > File "{PROJECT}/client.py", line 117, in test_stream for chunk in subgraph.stream( │ └ <function RemoteRunnable.stream at 0x7f765673ab60> └ <langserve.client.RemoteRunnable object at 0x7f7656751400> File "{PROJECT}/venv/lib/python3.13/site-packages/langserve/client.py", line 551, in stream chunk = self._lc_serializer.loads(sse["data"]) │ │ │ └ {'event': 'data', 'data': '{"search":{"query":"XXXXXXXXX",... │ │ └ <function Serializer.loads at 0x7f7656819ee0> │ └ <langserve.serialization.WellKnownLCSerializer object at 0x7f76565a82f0> └ <langserve.client.RemoteRunnable object at 0x7f7656751400> File "{PROJECT}/venv/lib/python3.13/site-packages/langserve/serialization.py", line 166, in loads return self.loadd(orjson.loads(s)) │ │ │ │ └ '{"search":{"query":"XXXXXXX","max_results... │ │ │ └ <built-in function loads> │ │ └ <module 'orjson' from '{PROJECT}/venv/lib/python3.13/site-packages/orjson/__init__.py'> │ └ <function WellKnownLCSerializer.loadd at 0x7f765681a200> └ <langserve.serialization.WellKnownLCSerializer object at 0x7f76565a82f0> orjson.JSONDecodeError: unexpected end of data: line 1 column 6604 (char 6603) ``` This is weird and we thought it makes no sense. You see, HTTP is TCP-based, so even though we don't have HTTPS set up for internal servers, **partial JSON** shouldn't appear in responses. We set up coredumpy and dowhen in our environment, and we did this. ```python from dowhen import when def _cb1(chunk: Any): logger.warning(chunk) when(langserve.APIHandler.stream, "yield {").do(_cb1) when(langserve.APIHandler.stream, "yield {").do('__import__("coredumpy").dump()') ``` Coredumpy is a post-mortem tool that stores the current frame state into a static dump file, so you can load and run a Pdb-like debugging console afterwards. Dowhen is a tool that injects a bit of code into a callable, so it's equivent to adding the code below the marked line (marked via the code keyword specified in `when()`). We did set it up for the `test_stream()` (in the test client) above, so we got the json data where an unexpected end of file appear. This way we got the error json happened in test_stream(), and the original state dumped as json (via orjson, same way as WellKnownLCSerializer) <img width="3808" height="1144" alt="Image" src="https://github.com/user-attachments/assets/7301a682-7dda-4744-8dcb-8f7876bbfc6b" /> AFAIK, Python's embedded `json.dumps` has a `ensure_ascii: bool` that prevents those characters from a serialized JSON string, but I didn't see one in orjson. We implement one with `json.dumps` instead, below: ```python class NotWellKnownSerializer(langserve.serialization.Serializer): def dumps(self, obj: Any) -> bytes: """Dump the given object to a JSON byte string.""" return json.dumps(obj, default=langserve.serialization.default, ensure_ascii=True).encode("utf-8") def loadd(self, s: Any) -> Any: return langserve.serialization._decode_lc_objects(s) ``` And we re-run our test_client, see it just works. Unfortunately I don't have screenshots for this, but it really just works fine then. The client gets the full fine response. I'm filing this issue since I'm not sure if this issue belongs to LangServe or FastAPI, or starlette_sse, so any guidance on whether this is expected behavior, a known limitation, or something that should be fixed at the LangServe level, would be much appreciated. Thanks for maintaining this project by the way!
yindo closed this issue 2026-02-16 00:19:49 -05:00
yindo commented 2026-02-16 00:19:49 -05:00
Author
Owner
Copy Link
Copy Source

@sorenchiron commented on GitHub (Sep 9, 2025):

same issue, fixed using this solution.
a proper json serializer shall be passed to remote runnable 'serializer' argument.

@sorenchiron commented on GitHub (Sep 9, 2025): same issue, fixed using this solution. a proper json serializer shall be passed to remote runnable 'serializer' argument.
yindo commented 2026-02-16 00:19:50 -05:00
Author
Owner
Copy Link
Copy Source

@fred913 commented on GitHub (Sep 9, 2025):

It turns out it's orjson preserving a \u2029 into JSON, and httpx thought it's a new line. So the \u2029 splits a json into N piecess, where none of the pieces are fine JSON. That's why replacing the orjson serializer with a json one works, since it now produces correct json without a new-line character inside.

I believe this issue now belongs to orjson.

@fred913 commented on GitHub (Sep 9, 2025): It turns out it's orjson preserving a \u2029 into JSON, and httpx thought it's a new line. So the \u2029 splits a json into N piecess, where none of the pieces are fine JSON. That's why replacing the orjson serializer with a json one works, since it now produces correct json without a new-line character inside. I believe this issue now belongs to orjson.
yindo commented 2026-02-16 00:19:50 -05:00
Author
Owner
Copy Link
Copy Source

@fred913 commented on GitHub (Sep 10, 2025):

If possible, could any of the maintainers please help me transfer this issue to https://github.com/ijl/orjson? I don't see such an option on my side 🤔

@fred913 commented on GitHub (Sep 10, 2025): If possible, could any of the maintainers please help me transfer this issue to https://github.com/ijl/orjson? I don't see such an option on my side 🤔
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
confirmed
duplicate
enhancement
help wanted
in-progress
investigate
not langserve
playground
pull-request
Mirrored from GitHub Pull Request
 question
stale
streaming
unable-to-reproduce
wontfix
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langchain-ai/langserve#252
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?