LangServe: 'invoke' Route Returns Empty Output, While Streaming Works #63

New Issue

Open

opened 2026-02-16 00:18:24 -05:00 by yindo · 13 comments

yindo commented 2026-02-16 00:18:24 -05:00

Owner

Copy Link

Copy Source

Originally created by @VML-Kai on GitHub (Dec 7, 2023).

Originally assigned to: @eyurtsev on GitHub.

I'm building a very simple LangChain application that takes as an input a customer feedback string and categorizes it into the following pydantic class:

class AnalysisAttributes(BaseModel):
    overall_positive: bool = Field(description="<sentiment is positive overall>")
    mentions_pricing: bool = Field(description="<pricing is mentioned>")
    mentions_competition: bool = Field(description="<competition is mentioned>")

parser = PydanticOutputParser(pydantic_object=AnalysisAttributes)

Here's how this should work, and it does:

full_pipeline = prompt | model | parser

output = full_pipeline.invoke({"feedback": "This bad company is very expensive."})

expected_output = AnalysisAttributes(overall_positive=False, mentions_pricing=True, mentions_competition=False)
assert output == expected_output.  # this works! :)

This works very well, all good so far! Let's serve it:

app = FastAPI(
  title="LangChain Server",
  version="1.0",
  description="A simple api server using Langchain's Runnable interfaces",
)

pipeline = prompt | model | parser
add_routes(app, pipeline, path="/categorize_feedback")

if __name__ == "__main__":
    import uvicorn
    uvicorn.run(app, host="localhost", port=8000)

Now comes the strange part, check this out. On the client side, streaming works:

response = requests.post(
    "http://localhost:8000/categorize_feedback/stream/",
    json={'input': {'feedback': 'Prices are too high.'}}
)
for chunk in response:
    print(chunk.decode())

# event: metadata [...] data: {"overall_positive":false, ...

But the regular invoke does not work, it delivers an empty output:

response = requests.post(
    "http://localhost:8000/categorize_feedback/invoke/",
    json={'input': {'feedback': 'Prices are too high.'}}
)
print(response.json())

# {'output': {}, 'callback_events': [], 'metadata': {'run_id': 'acdd089d-3c80-4624-8122-17c4173dc1ec'}}

I'm pretty sure the problem is the output parser, as if I remove it from the chain, everything works perfectly fine, the invoke as well.

Originally created by @VML-Kai on GitHub (Dec 7, 2023). Originally assigned to: @eyurtsev on GitHub. I'm building a very simple LangChain application that takes as an input a customer feedback string and categorizes it into the following pydantic class: ``` class AnalysisAttributes(BaseModel): overall_positive: bool = Field(description="<sentiment is positive overall>") mentions_pricing: bool = Field(description="<pricing is mentioned>") mentions_competition: bool = Field(description="<competition is mentioned>") parser = PydanticOutputParser(pydantic_object=AnalysisAttributes) ``` Here's how this should work, and it does: ``` full_pipeline = prompt | model | parser output = full_pipeline.invoke({"feedback": "This bad company is very expensive."}) expected_output = AnalysisAttributes(overall_positive=False, mentions_pricing=True, mentions_competition=False) assert output == expected_output. # this works! :) ``` This works very well, all good so far! Let's serve it: ``` app = FastAPI( title="LangChain Server", version="1.0", description="A simple api server using Langchain's Runnable interfaces", ) pipeline = prompt | model | parser add_routes(app, pipeline, path="/categorize_feedback") if __name__ == "__main__": import uvicorn uvicorn.run(app, host="localhost", port=8000) ``` Now comes the strange part, check this out. On the client side, streaming works: ``` response = requests.post( "http://localhost:8000/categorize_feedback/stream/", json={'input': {'feedback': 'Prices are too high.'}} ) for chunk in response: print(chunk.decode()) # event: metadata [...] data: {"overall_positive":false, ... ``` But the regular invoke does not work, it delivers an empty output: ``` response = requests.post( "http://localhost:8000/categorize_feedback/invoke/", json={'input': {'feedback': 'Prices are too high.'}} ) print(response.json()) # {'output': {}, 'callback_events': [], 'metadata': {'run_id': 'acdd089d-3c80-4624-8122-17c4173dc1ec'}} ``` I'm pretty sure the problem is the output parser, as if I remove it from the chain, everything works perfectly fine, the invoke as well.

yindo added the bug label 2026-02-16 00:18:24 -05:00

yindo commented 2026-02-16 00:18:24 -05:00

Author

Owner

Copy Link

Copy Source

@VML-Kai commented on GitHub (Dec 8, 2023):

Currently using a workaround using FastAPI directly:

@app.post('/categorize_feedback')
async def categorize_feedback(input_data: FeedbackInput) -> AnalysisAttributes:
    full_pipeline = prompt | model | parser
    output = full_pipeline.invoke(input_data.model_dump())
    return output

I think somehow .invoke isn't being called differently in langserve.

@VML-Kai commented on GitHub (Dec 8, 2023): Currently using a workaround using FastAPI directly: ``` @app.post('/categorize_feedback') async def categorize_feedback(input_data: FeedbackInput) -> AnalysisAttributes: full_pipeline = prompt | model | parser output = full_pipeline.invoke(input_data.model_dump()) return output ``` I think somehow .invoke isn't being called differently in langserve.

yindo commented 2026-02-16 00:18:25 -05:00

Author

Owner

Copy Link

Copy Source

@eyurtsev commented on GitHub (Dec 8, 2023):

Hello @VML-Kai , thanks for the bug report.

Would you be able to tell me:

1. What are your langserve and langchain versions?
2. Which version of pydantic are you using

---

I'm wondering is whether the automatic type inference is failing when the parser is present and the chain is doing something weird.

If you're able to:

3. Could you share the input and output types of runnable chain? (chain.input_schema.schema() i think or get the jsonschema for those?)
4. Could you manually overload the input/output types:

    def with_types(
        self,
        input_type: Optional[Union[Type[Input], BaseModel]] = None,
        output_type: Optional[Union[Type[Output], BaseModel]] = None,
    ) -> Runnable[Input, Output]:

@eyurtsev commented on GitHub (Dec 8, 2023): Hello @VML-Kai , thanks for the bug report. Would you be able to tell me: 1) What are your langserve and langchain versions? 2) Which version of pydantic are you using --- I'm wondering is whether the automatic type inference is failing when the parser is present and the chain is doing something weird. If you're able to: 3) Could you share the input and output types of runnable chain? (chain.input_schema.schema() i think or get the jsonschema for those?) 4) Could you manually overload the input/output types: ```python def with_types( self, input_type: Optional[Union[Type[Input], BaseModel]] = None, output_type: Optional[Union[Type[Output], BaseModel]] = None, ) -> Runnable[Input, Output]: ```

yindo commented 2026-02-16 00:18:25 -05:00

Author

Owner

Copy Link

Copy Source

@eyurtsev commented on GitHub (Dec 8, 2023):

@VML-Kai I noticed that the URLs that you have a trailing slash

response = requests.post(
    "http://localhost:8000/categorize_feedback/invoke/",
    json={'input': {'feedback': 'Prices are too high.'}}
)```

Could you remove the trailing slash and try again?

```python
response = requests.post(
    "http://localhost:8000/categorize_feedback/invoke",
    json={'input': {'feedback': 'Prices are too high.'}}
)

@eyurtsev commented on GitHub (Dec 8, 2023): @VML-Kai I noticed that the URLs that you have a trailing slash ```python response = requests.post( "http://localhost:8000/categorize_feedback/invoke/", json={'input': {'feedback': 'Prices are too high.'}} )``` Could you remove the trailing slash and try again? ```python response = requests.post( "http://localhost:8000/categorize_feedback/invoke", json={'input': {'feedback': 'Prices are too high.'}} ) ```

yindo commented 2026-02-16 00:18:25 -05:00

Author

Owner

Copy Link

Copy Source

@danielcj commented on GitHub (Dec 17, 2023):

@VML-Kai I was experiencing the same thing, but it works as expected when declaring the output type.

Example on your code:

pipeline = prompt | model | parser
add_routes(app, pipeline, path="/categorize_feedback", output_type=AnalysisAttributes)

@danielcj commented on GitHub (Dec 17, 2023): @VML-Kai I was experiencing the same thing, but it works as expected when declaring the output type. Example on your code: > ``` > pipeline = prompt | model | parser > add_routes(app, pipeline, path="/categorize_feedback", output_type=AnalysisAttributes) > ```

yindo commented 2026-02-16 00:18:25 -05:00

Author

Owner

Copy Link

Copy Source

@eyurtsev commented on GitHub (Dec 21, 2023):

@danielcj thank you that's helpful! I'll try to reproduce!

@eyurtsev commented on GitHub (Dec 21, 2023): @danielcj thank you that's helpful! I'll try to reproduce!

yindo commented 2026-02-16 00:18:25 -05:00

Author

Owner

Copy Link

Copy Source

@effusive-ai commented on GitHub (Jan 25, 2024):

I think I'm having this same problem but after some more investigation I'm not sure. Just a simple chain like this

template = "Write a summary of the following article: \"{content}\""
prompt = PromptTemplate(template=template, input_variables=["content"])

add_routes(
    app,
    prompt | llm,
    path="/summary",
)

I'm calling the route from the JS RemoteRunable class. I thought it was just invoke that wasn't working. But now I see both invoke and stream sometimes return a result and sometimes an empty string for the same call. I haven't been able to track down a pattern of why one call works and another doesn't. Just installed langserv yesterday so should be the latest version.

@effusive-ai commented on GitHub (Jan 25, 2024): I think I'm having this same problem but after some more investigation I'm not sure. Just a simple chain like this ``` template = "Write a summary of the following article: \"{content}\"" prompt = PromptTemplate(template=template, input_variables=["content"]) add_routes( app, prompt | llm, path="/summary", ) ``` I'm calling the route from the JS RemoteRunable class. I thought it was just invoke that wasn't working. But now I see both invoke and stream sometimes return a result and sometimes an empty string for the same call. I haven't been able to track down a pattern of why one call works and another doesn't. Just installed langserv yesterday so should be the latest version.

yindo commented 2026-02-16 00:18:26 -05:00

Author

Owner

Copy Link

Copy Source

@PhillipsHoward commented on GitHub (Feb 14, 2024):

Hello all.
I have the same issue I think :

Defining this on server side :

(...)
agent = create_openai_functions_agent(llm, tools, prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)

# Chat history
config={"configurable": {"session_id": "test_session"}}
agent_with_history = RunnableWithMessageHistory(
    agent_executor,
    lambda session_id: RedisChatMessageHistory(session_id, url=os.getenv('REDIS_URL')),
    input_messages_key="input",
    history_messages_key="chat_history",
)

app = FastAPI(
  title="LangChain Test Server",
  version="1.0",
  description="A simple API server using LangChain",
)

add_routes(
    app,
    agent_with_history,
    path="/narrator",
)

Then on my "client" side, I try :

def main():
    chat = RemoteRunnable("http://localhost:8000/narrator")
    session_id = generate_random_string(10)
    
    while True:
        human_input = input("You: ")  # Get user input
        response = chat.invoke({'input': human_input}, {'configurable': {'session_id': session_id}})
        print(response)

The response is always an empty {}, while logs in server indicates that everything works fine on server side. (I can see the output displayed in it)

My pip list about this :

langchain                 0.1.5
langchain-cli             0.0.21
langchain-community       0.0.17
langchain-core            0.1.18
langchain-openai          0.0.5
langserve                 0.0.41
fastapi                   0.109.2

Am I missing something ?
Thanks in advance !

EDIT :

As @VML-Kai suggested, it works fine with this workaround :

@app.post('/narrator')
async def narrator_chat(input_data : InputData):
    output = narrator_chain.invoke({'input': input_data.input}, {'configurable': {'session_id': input_data.session_id}})
    return output

@PhillipsHoward commented on GitHub (Feb 14, 2024): Hello all. I have the same issue I think : Defining this on server side : ```python (...) agent = create_openai_functions_agent(llm, tools, prompt) agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True) # Chat history config={"configurable": {"session_id": "test_session"}} agent_with_history = RunnableWithMessageHistory( agent_executor, lambda session_id: RedisChatMessageHistory(session_id, url=os.getenv('REDIS_URL')), input_messages_key="input", history_messages_key="chat_history", ) app = FastAPI( title="LangChain Test Server", version="1.0", description="A simple API server using LangChain", ) add_routes( app, agent_with_history, path="/narrator", ) ``` Then on my "client" side, I try : ```python def main(): chat = RemoteRunnable("http://localhost:8000/narrator") session_id = generate_random_string(10) while True: human_input = input("You: ") # Get user input response = chat.invoke({'input': human_input}, {'configurable': {'session_id': session_id}}) print(response) ``` The response is always an empty {}, while logs in server indicates that everything works fine on server side. (I can see the output displayed in it) My pip list about this : ``` langchain 0.1.5 langchain-cli 0.0.21 langchain-community 0.0.17 langchain-core 0.1.18 langchain-openai 0.0.5 langserve 0.0.41 fastapi 0.109.2 ``` Am I missing something ? Thanks in advance ! EDIT : As @VML-Kai suggested, it works fine with this workaround : ```python @app.post('/narrator') async def narrator_chat(input_data : InputData): output = narrator_chain.invoke({'input': input_data.input}, {'configurable': {'session_id': input_data.session_id}}) return output ```

yindo commented 2026-02-16 00:18:26 -05:00

Author

Owner

Copy Link

Copy Source

@realei commented on GitHub (Apr 29, 2024):

I have the same issue when I wrap Agent as a Langserve endpoint with invoke endpoint

my pip list:

langchain==0.1.14
langchain-cli==0.0.21
langchain-community==0.0.31
langchain-core==0.1.40
langchain-openai==0.1.1
langchain-text-splitters==0.0.1
langserve==0.0.51
langsmith==0.1.40
fastapi==0.110

@realei commented on GitHub (Apr 29, 2024): I have the same issue when I wrap Agent as a Langserve endpoint with `invoke` endpoint my pip list: ``` langchain==0.1.14 langchain-cli==0.0.21 langchain-community==0.0.31 langchain-core==0.1.40 langchain-openai==0.1.1 langchain-text-splitters==0.0.1 langserve==0.0.51 langsmith==0.1.40 fastapi==0.110 ```

yindo commented 2026-02-16 00:18:26 -05:00

Author

Owner

Copy Link

Copy Source

@Aillian commented on GitHub (May 22, 2024):

is this issue solved? i am facing the same problem

@Aillian commented on GitHub (May 22, 2024): is this issue solved? i am facing the same problem

yindo commented 2026-02-16 00:18:26 -05:00

Author

Owner

Copy Link

Copy Source

@shreyasjha5706 commented on GitHub (May 28, 2024):

i had the same issue, @danielcj 's solution worked for me. Thanks!

@shreyasjha5706 commented on GitHub (May 28, 2024): i had the same issue, @danielcj 's solution worked for me. Thanks!

yindo commented 2026-02-16 00:18:26 -05:00

Author

Owner

Copy Link

Copy Source

@MarkEdmondson1234 commented on GitHub (Jul 10, 2024):

Same issue today with versions:

langchain                      0.2.6
langserve                      0.2.2

/stream endpoint works, /invoke to same chain has empty content:

{"output":{},"metadata":{"run_id":"9c0f41e4-086d-49f3-9d6b-13352e26e6b2","feedback_tokens":[]}}

Tried adding Output type, no change:

class Output(BaseModel):
    content: str
    metadata: Optional[Dict]

chain = chain.with_types(input_type=Question, output_type=Output).pick(["content", "metadata"])

@MarkEdmondson1234 commented on GitHub (Jul 10, 2024): Same issue today with versions: ``` langchain 0.2.6 langserve 0.2.2 ``` /stream endpoint works, /invoke to same chain has empty content: ``` {"output":{},"metadata":{"run_id":"9c0f41e4-086d-49f3-9d6b-13352e26e6b2","feedback_tokens":[]}} ``` Tried adding Output type, no change: ```python class Output(BaseModel): content: str metadata: Optional[Dict] chain = chain.with_types(input_type=Question, output_type=Output).pick(["content", "metadata"]) ```

yindo commented 2026-02-16 00:18:26 -05:00

Author

Owner

Copy Link

Copy Source

@manova commented on GitHub (Jul 18, 2024):

I'm having the same issue where stream returns a response but the invoke endpoint returns null. Are there any new findings as to the root cause of this?

@manova commented on GitHub (Jul 18, 2024): I'm having the same issue where stream returns a response but the invoke endpoint returns null. Are there any new findings as to the root cause of this?

yindo commented 2026-02-16 00:18:27 -05:00

Author

Owner

Copy Link

Copy Source

@MarkEdmondson1234 commented on GitHub (Jul 18, 2024):

Mine got fixed putting the output class on the app function (eg making the endpoints) not the chain

@MarkEdmondson1234 commented on GitHub (Jul 18, 2024): Mine got fixed putting the output class on the app function (eg making the endpoints) not the chain

yindo referenced this issue 2026-02-16 00:20:08 -05:00

[PR #63] [MERGED] Playground Cleanup #318

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

dependabot/pip/pip-82fdfb001f

dependabot/pip/jupyterlab-4.5.6

dependabot/pip/fastapi-0.136.0

dependabot/pip/pydantic-2.13.3

dependabot/pip/sse-starlette-3.3.4

dependabot/pip/pytest-asyncio-1.3.0

sam/bump-langchain

joaquin/test-pushing-sk-to-langserver-should-block

joaquin/bump-again-due-to-failed-release

eyurtsev-patch-5

wfh/relax_sse_starlette

eyurtsev-patch-4

jacob/docs

nc/22apr/serve

add-playground-test

eugene/playground_security

eyurtsev-patch-3

jacob/chat_widget

eyurtsev-patch-2

eyurtsev-patch-1

eugene/add_is_instance_check

eugene/agent_executor_playground

revert-209-2023-11-10-format-langsmith-runs

2023-11-10-format-langsmith-runs

harrison/dont-do-validation

nc/4nov

eugene/playground

eugene/exmaples

dqbd/playground-autotype-tuple-fix

erick/packages-bugfix

erick/assign-input-params

jacob/playground

jacob/allowlist

erick/langservehub

eugene/add_interpreptors

v0.3.3

v0.3.2

v0.3.1

v0.3.0

v0.3.0dev2

v0.3.0dev1

v0.2.3

v0.2.2

v0.2.1

v0.2.0

v0.2.0rc1

v0.1.1

v0.1.0

v0.1.0rc2

v0.1.0rc1

v0.0.51

v0.0.50

v0.0.49

v0.0.48

v0.0.47

v0.0.46

v0.0.45

v0.0.44

v0.0.43

v0.0.42

v0.0.41

v0.0.40

v0.0.39

v0.0.38

v0.0.37

v0.0.36

v0.0.35

v0.0.34

v0.0.33

v0.0.32

v0.0.31

v0.0.30

v0.0.29

v0.0.28

v0.0.27

v0.0.26

v0.0.25

v0.0.24

v0.0.23

v0.0.22

v0.0.21

v0.0.20

v0.0.19

v0.0.18

v0.0.17

v0.0.16

v0.0.15

v0.0.14

v0.0.13

v0.0.12

v0.0.11

v0.0.10

v0.0.9

v0.0.8

v0.0.7

v0.0.6

v0.0.5

v0.0.4

v0.0.3

v0.0.2

Labels

Clear labels

bug

confirmed

duplicate

enhancement

help wanted

in-progress

investigate

not langserve

playground

pull-request

Mirrored from GitHub Pull Request

question

stale

streaming

unable-to-reproduce

wontfix

No Label bug

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

yindo

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: langchain-ai/langserve#63