mirror of
https://github.com/langchain-ai/langserve.git
synced 2026-07-19 22:03:48 -04:00
Feature Request: Serve LangChain as an OpenAI API Compatible Server #92
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @2catycm on GitHub (Jan 14, 2024).
In many LLM Application, OpenAI API is a widely used format. These applications directly use ChatGPT via api key and openai client library. After that, they are empowered by LLM and have there functions.
It is a common need to change the LLM (e.g. change to LangChain/RAG enabled LLM; or just change to another LLM), but it is not easy to change the code in these applications.
We can run a server locally that is OpenAI API compatible, and change the OPENAI_BASE_URL env variable that the LLM application would use.
Can LangServe do that?
@2catycm commented on GitHub (Jan 16, 2024):
For the langchain applications with input and output being both of single str type, this is possible.
@eyurtsev commented on GitHub (Jan 16, 2024):
LangChain provides a compatibility layer over all chat models regardless of provider since all chat models follow more or less the same API.
Here's a server that allows serving both Anthropic and OpenAI models: https://github.com/langchain-ai/langserve/blob/main/examples/llm/server.py
Use Configurable Runnables to allow run time configuration of the chat model; e.g., that could be used with OpenAI chat models to control parameters like temperature or model name based on information provided on the client side.
You can use configurable runnables to select between chat models with entirely different underlying APIs; e.g., Anthropic and control which model is used using run time parameters as well.
Hope this helps.
@eyurtsev commented on GitHub (Jan 16, 2024):
The RemoteRunnable client takes care of serialization of well known LangChain primitives like AIMessage / HumanMessage / FunctionMessage etc (so it works with non string types out of the box)
@2catycm commented on GitHub (Jan 16, 2024):
That is great. LangChain and LangServe based software can use the same batch/stream/async API while having different configuration via Configurable Runnable with a variety of LLMs.
But I am considering programs that are not LangChain based. These programs may use openai with the openai python lib directly. I am considering to make a proxy server that is compatible with OpenAI, but it actually invokes LangChain developped Agents.
@2catycm commented on GitHub (Jan 16, 2024):
That's cool
@eyurtsev commented on GitHub (Jan 16, 2024):
Could you elaborate?
It sounds like the client code is not langchain based, but the server code is langchain based (since it's running a langchain agent?) Is that the scenario you're thinking about?
@2catycm commented on GitHub (Jan 17, 2024):
Yes, LangChain Agent as a Model as a Service. which is what langserve is doing.
langserve's API has its format as indicated in langserve documentation.
Application based on LLM are too assorted to enumerate. They are of various kind. Not all of them have leveraged LangChain, while they all must at least support ChatGPT API, which is of https://platform.openai.com/docs/api-reference/chat/create?lang=python format
@eyurtsev commented on GitHub (Jan 27, 2024):
Got it -- LangServe won't be helpful for this use case since it's optimized for serving LangChain runnables
Creating an Open AI like API sounds pretty straight forward using vanilla LangChain and Fast API. LangChain provides a uniform interface over a lot of LLMs, so you would need to write an adapter between Open AI like interface and the LangChain runnable interface. (This is not a feature we're likely to develop since we haven't seen many users request it and I suspect that there are already 3rd party providers that do exactly that.)
@2catycm commented on GitHub (Jan 27, 2024):
got it, thanks for your detailed reply
@PlebeiusGaragicus commented on GitHub (Jul 2, 2024):
Came here wondering the same.
I think a good example use case is a project like Open WebUI. It can easily serve Ollama models as well as connect to OpenAI compatible APIs.
But, there's no easy way to incorporate LangServe. LangServe gives the typical /stream, /batch, /astream_events, etc endpoints.... but these aren't OpenAI compatible and thus I need to find work-arounds and add layers in order to get this to work.
@Schumacher-Bastian commented on GitHub (Sep 18, 2024):
Yes, this would be a great feature.
@reteps commented on GitHub (Sep 25, 2024):
I found this: https://github.com/samuelint/langchain-openai-api-bridge which seems to fit my usecase.
@2catycm commented on GitHub (Sep 26, 2024):
Looks great!