Update docs links

Use correct attribute

.clabot

@@ -1,4 +1,4 @@
 {
-  "contributors": ["eyurtsev", "hwchase17", "nfcampos", "efriis", "jacoblee93", "dqbd", "kreneskyp", "adarsh-jha-dev", "harris", "baskaryan", "hinthornw", "bracesproul", "jakerachleff", "craigsdennis", "anhi", "169", "LarchLiu", "PaulLockett", "RCMatthias", "jwynia", "majiayu000", "mpskex", "shivachittamuru", "sinashaloudegi", "sowsan", "akira", "lucianotonet", "JGalego", "nat-n", "dirien"],
+  "contributors": ["eyurtsev", "hwchase17", "nfcampos", "efriis", "jacoblee93", "dqbd", "kreneskyp", "adarsh-jha-dev", "harris", "baskaryan", "hinthornw", "bracesproul", "jakerachleff", "craigsdennis", "anhi", "169", "LarchLiu", "PaulLockett", "RCMatthias", "jwynia", "majiayu000", "mpskex", "shivachittamuru", "sinashaloudegi", "sowsan", "akira", "lucianotonet", "JGalego", "nat-n", "dirien", "donbr", "rahilvora", "WarrenTheRabbit", "StreetLamb", "ccurme"],
   "message": "Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have the username {{usersWithoutCLA}} on file. In order for us to review and merge your code, please complete the Individual Contributor License Agreement here https://forms.gle/AQFbtkWRoHXUgipM6 .\n\nThis process is done manually on our side, so after signing the form one of the maintainers will add you to the contributors list.\n\nFor more details about why we have a CLA and other contribution guidelines please see: https://github.com/langchain-ai/langserve/blob/main/CONTRIBUTING.md."
 }

CONTRIBUTING.md

@@ -49,3 +49,32 @@ To run linting for this project:
 ```sh
 make lint
 ```
+
+## Frontend Playground Development
+
+Here are a few tips to keep in mind when developing the LangServe playgrounds:
+
+### Setup
+
+Switch directories to `langserve/playground` or `langserve/chat_playground`, then run `yarn` to install required
+dependencies. `yarn dev` will start the playground at `http://localhost:5173/____LANGSERVE_BASE_URL/` in dev mode.
+
+You can run one of the chains in the `examples/` repo using `poetry run python path/to/file.py`.
+
+### Setting CORS
+
+You may need to add the following to an example route when developing the playground in dev mode to handle CORS:
+
+```python
+from fastapi.middleware.cors import CORSMiddleware
+
+# Set all CORS enabled origins
+app.add_middleware(
+    CORSMiddleware,
+    allow_origins=["*"],
+    allow_credentials=True,
+    allow_methods=["*"],
+    allow_headers=["*"],
+    expose_headers=["*"],
+)
+```

README.md

@@ -6,13 +6,13 @@
 [![](https://dcbadge.vercel.app/api/server/6adMQxSpJS?compact=true&style=flat)](https://discord.com/channels/1038097195422978059/1170024642245832774)
 
 🚩 We will be releasing a hosted version of LangServe for one-click deployments of
-LangChain applications. [Sign up here](https://airtable.com/app0hN6sd93QcKubv/shrAjst60xXa6quV2)
+LangChain applications. [Sign up here](https://forms.gle/KC13Nzn76UeLaghK7)
 to get on the waitlist.
 
 ## Overview
 
 [LangServe](https://github.com/langchain-ai/langserve) helps developers
-deploy `LangChain` [runnables and chains](https://python.langchain.com/docs/expression_language/)
+deploy `LangChain` [runnables and chains](https://python.langchain.com/v0.2/docs/how_to/#langchain-expression-language-lcel)
 as a REST API.
 
 This library is integrated with [FastAPI](https://fastapi.tiangolo.com/) and
@@ -28,11 +28,11 @@ in [LangChain.js](https://js.langchain.com/docs/ecosystem/langserve).
 - Input and Output schemas automatically inferred from your LangChain object, and
   enforced on every API call, with rich error messages
 - API docs page with JSONSchema and Swagger (insert example link)
-- Efficient `/invoke/`, `/batch/` and `/stream/` endpoints with support for many
+- Efficient `/invoke`, `/batch` and `/stream` endpoints with support for many
   concurrent requests on a single server
-- `/stream_log/` endpoint for streaming all (or some) intermediate steps from your
+- `/stream_log` endpoint for streaming all (or some) intermediate steps from your
   chain/agent
-- **new** as of 0.0.40, supports `astream_events` to make it easier to stream without needing to parse the output of `stream_log`.
+- **new** as of 0.0.40, supports `/stream_events` to make it easier to stream without needing to parse the output of `/stream_log`.
 - Playground page at `/playground/` with streaming output and intermediate steps
 - Built-in (optional) tracing to [LangSmith](https://www.langchain.com/langsmith), just
   add your API key (see [Instructions](https://docs.smith.langchain.com/))
@@ -53,12 +53,12 @@ in [LangChain.js](https://js.langchain.com/docs/ecosystem/langserve).
 
 We will be releasing a hosted version of LangServe for one-click deployments of
 LangChain
-applications. [Sign up here](https://airtable.com/apppQ9p5XuujRl3wJ/shrABpHWdxry8Bacm)
+applications. [Sign up here](https://forms.gle/KC13Nzn76UeLaghK7)
 to get on the waitlist.
 
 ## Security
 
-* Vulnerability in Versions 0.0.13 - 0.0.15 -- playground endpoint allows accessing
+- Vulnerability in Versions 0.0.13 - 0.0.15 -- playground endpoint allows accessing
   arbitrary files on
   server. [Resolved in 0.0.16](https://github.com/langchain-ai/langserve/pull/98).
 
@@ -80,8 +80,38 @@ Use the `LangChain` CLI to bootstrap a `LangServe` project quickly.
 To use the langchain CLI make sure that you have a recent version of `langchain-cli`
 installed. You can install it with `pip install -U langchain-cli`.
 
+## Setup
+
+**Note**: We use `poetry` for dependency management. Please follow poetry [doc](https://python-poetry.org/docs/) to learn more about it.
+
+### 1. Create new app using langchain cli command
+
 ```sh
-langchain app new ../path/to/directory
+langchain app new my-app
+```
+
+### 2. Define the runnable in add_routes. Go to server.py and edit
+
+```sh
+add_routes(app. NotImplemented)
+```
+
+### 3. Use `poetry` to add 3rd party packages (e.g., langchain-openai, langchain-anthropic, langchain-mistral etc).
+
+```sh
+poetry add [package-name] // e.g `poetry add langchain-openai`
+```
+
+### 4. Set up relevant env variables. For example,
+
+```sh
+export OPENAI_API_KEY="sk-..."
+```
+
+### 5. Serve your app
+
+```sh
+poetry run langchain serve --port=8100
 ```
 
 ## Examples
@@ -95,24 +125,24 @@ or the [examples](https://github.com/langchain-ai/langserve/tree/main/examples)
 directory.
 
 | Description                                                                                                                                                                                                                                                        | Links                                                                                                                                                                                                                               |
-|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| **LLMs** Minimal example that reserves OpenAI and Anthropic chat models. Uses async, supports batching and streaming.                                                                                                                                              | [server](https://github.com/langchain-ai/langserve/tree/main/examples/llm/server.py), [client](https://github.com/langchain-ai/langserve/blob/main/examples/llm/client.ipynb)                                                       | 
-| **Retriever** Simple server that exposes a retriever as a runnable.                                                                                                                                                                                                | [server](https://github.com/langchain-ai/langserve/tree/main/examples/retrieval/server.py), [client](https://github.com/langchain-ai/langserve/tree/main/examples/retrieval/client.ipynb)                                           | 
-| **Conversational Retriever** A [Conversational Retriever](https://python.langchain.com/docs/expression_language/cookbook/retrieval#conversational-retrieval-chain) exposed via LangServe                                                                           | [server](https://github.com/langchain-ai/langserve/tree/main/examples/conversational_retrieval_chain/server.py), [client](https://github.com/langchain-ai/langserve/tree/main/examples/conversational_retrieval_chain/client.ipynb) | 
-| **Agent** without **conversation history** based on [OpenAI tools](https://python.langchain.com/docs/modules/agents/agent_types/openai_functions_agent)                                                                                             | [server](https://github.com/langchain-ai/langserve/tree/main/examples/agent/server.py), [client](https://github.com/langchain-ai/langserve/tree/main/examples/agent/client.ipynb)                                                   | 
-| **Agent** with **conversation history** based on [OpenAI tools](https://python.langchain.com/docs/modules/agents/agent_types/openai_functions_agent)                                                                                                               | [server](https://github.com/langchain-ai/langserve/blob/main/examples/agent_with_history/server.py), [client](https://github.com/langchain-ai/langserve/blob/main/examples/agent_with_history/client.ipynb)                         | 
-| [RunnableWithMessageHistory](https://python.langchain.com/docs/expression_language/how_to/message_history) to implement chat persisted on backend, keyed off a `session_id` supplied by client.                                                                    | [server](https://github.com/langchain-ai/langserve/tree/main/examples/chat_with_persistence/server.py), [client](https://github.com/langchain-ai/langserve/tree/main/examples/chat_with_persistence/client.ipynb)                   |
-| [RunnableWithMessageHistory](https://python.langchain.com/docs/expression_language/how_to/message_history) to implement chat persisted on backend, keyed off a `conversation_id` supplied by client, and `user_id` (see Auth for implementing `user_id` properly). | [server](https://github.com/langchain-ai/langserve/tree/main/examples/chat_with_persistence_and_user/server.py), [client](https://github.com/langchain-ai/langserve/tree/main/examples/chat_with_persistence_and_user/client.ipynb) | 
-| [Configurable Runnable](https://python.langchain.com/docs/expression_language/how_to/configure) to create a retriever that supports run time configuration of the index name.                                                                                      | [server](https://github.com/langchain-ai/langserve/tree/main/examples/configurable_retrieval/server.py), [client](https://github.com/langchain-ai/langserve/tree/main/examples/configurable_retrieval/client.ipynb)                 | 
-| [Configurable Runnable](https://python.langchain.com/docs/expression_language/how_to/configure) that shows configurable fields and configurable alternatives.                                                                                                      | [server](https://github.com/langchain-ai/langserve/tree/main/examples/configurable_chain/server.py), [client](https://github.com/langchain-ai/langserve/tree/main/examples/configurable_chain/client.ipynb)                         | 
+| :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| **LLMs** Minimal example that reserves OpenAI and Anthropic chat models. Uses async, supports batching and streaming.                                                                                                                                              | [server](https://github.com/langchain-ai/langserve/tree/main/examples/llm/server.py), [client](https://github.com/langchain-ai/langserve/blob/main/examples/llm/client.ipynb)                                                       |
+| **Retriever** Simple server that exposes a retriever as a runnable.                                                                                                                                                                                                | [server](https://github.com/langchain-ai/langserve/tree/main/examples/retrieval/server.py), [client](https://github.com/langchain-ai/langserve/tree/main/examples/retrieval/client.ipynb)                                           |
+| **Conversational Retriever** A Conversational Retriever exposed via LangServe                                                                           | [server](https://github.com/langchain-ai/langserve/tree/main/examples/conversational_retrieval_chain/server.py), [client](https://github.com/langchain-ai/langserve/tree/main/examples/conversational_retrieval_chain/client.ipynb) |
+| **Agent** without **conversation history** based on [OpenAI tools](https://python.langchain.com/docs/modules/agents/agent_types/openai_functions_agent)                                                                                                            | [server](https://github.com/langchain-ai/langserve/tree/main/examples/agent/server.py), [client](https://github.com/langchain-ai/langserve/tree/main/examples/agent/client.ipynb)                                                   |
+| **Agent** with **conversation history** based on [OpenAI tools](https://python.langchain.com/docs/modules/agents/agent_types/openai_functions_agent)                                                                                                               | [server](https://github.com/langchain-ai/langserve/blob/main/examples/agent_with_history/server.py), [client](https://github.com/langchain-ai/langserve/blob/main/examples/agent_with_history/client.ipynb)                         |
+| [RunnableWithMessageHistory](https://python.langchain.com/v0.2/docs/how_to/message_history/) to implement chat persisted on backend, keyed off a `session_id` supplied by client.                                                                    | [server](https://github.com/langchain-ai/langserve/tree/main/examples/chat_with_persistence/server.py), [client](https://github.com/langchain-ai/langserve/tree/main/examples/chat_with_persistence/client.ipynb)                   |
+| [RunnableWithMessageHistory](https://python.langchain.com/v0.2/docs/how_to/message_history/) to implement chat persisted on backend, keyed off a `conversation_id` supplied by client, and `user_id` (see Auth for implementing `user_id` properly). | [server](https://github.com/langchain-ai/langserve/tree/main/examples/chat_with_persistence_and_user/server.py), [client](https://github.com/langchain-ai/langserve/tree/main/examples/chat_with_persistence_and_user/client.ipynb) |
+| [Configurable Runnable](https://python.langchain.com/v0.2/docs/how_to/configure/) to create a retriever that supports run time configuration of the index name.                                                                                      | [server](https://github.com/langchain-ai/langserve/tree/main/examples/configurable_retrieval/server.py), [client](https://github.com/langchain-ai/langserve/tree/main/examples/configurable_retrieval/client.ipynb)                 |
+| [Configurable Runnable](https://python.langchain.com/v0.2/docs/how_to/configure/) that shows configurable fields and configurable alternatives.                                                                                                      | [server](https://github.com/langchain-ai/langserve/tree/main/examples/configurable_chain/server.py), [client](https://github.com/langchain-ai/langserve/tree/main/examples/configurable_chain/client.ipynb)                         |
 | **APIHandler** Shows how to use `APIHandler` instead of `add_routes`. This provides more flexibility for developers to define endpoints. Works well with all FastAPI patterns, but takes a bit more effort.                                                        | [server](https://github.com/langchain-ai/langserve/tree/main/examples/api_handler_examples/server.py)                                                                                                                               |
-| **LCEL Example** Example that uses LCEL to manipulate a dictionary input.                                                                                                                                                                                          | [server](https://github.com/langchain-ai/langserve/tree/main/examples/passthrough_dict/client.ipynb), [client](https://github.com/langchain-ai/langserve/tree/main/examples/passthrough_dict/client.ipynb)                          | 
-| **Auth** with `add_routes`: Simple authentication that can be applied across all endpoints associated with app. (Not useful on its own for implementing per user logic.)                                                                                           | [server](https://github.com/langchain-ai/langserve/tree/main/examples/auth/global_deps/server.py)                                                                                                                                   | 
-| **Auth** with `add_routes`: Simple authentication mechanism based on path dependencies. (No useful on its own for implementing per user logic.)                                                                                                                    | [server](https://github.com/langchain-ai/langserve/tree/main/examples/auth/path_dependencies/server.py)                                                                                                                             | 
-| **Auth** with `add_routes`: Implement per user logic and auth for endpoints that use per request config modifier. (**Note**: At the moment, does not integrate with OpenAPI docs.)                                                                                 | [server](https://github.com/langchain-ai/langserve/tree/main/examples/auth/per_req_config_modifier/server.py), [client](https://github.com/langchain-ai/langserve/tree/main/examples/auth/per_req_config_modifier/client.ipynb)     | 
-| **Auth** with `APIHandler`: Implement per user logic and auth that shows how to search only within user owned documents.                                                                                                                                           | [server](https://github.com/langchain-ai/langserve/tree/main/examples/auth/api_handler/server.py), [client](https://github.com/langchain-ai/langserve/tree/main/examples/auth/api_handler/client.ipynb)                             | 
-| **Widgets** Different widgets that can be used with playground (file upload and chat)                                                                                                                                                                              | [server](https://github.com/langchain-ai/langserve/tree/main/examples/widgets/chat/tuples/server.py)                                                                                                                                            | 
-| **Widgets** File upload widget used for LangServe playground.                                                                                                                                                                                                      | [server](https://github.com/langchain-ai/langserve/tree/main/examples/file_processing/server.py), [client](https://github.com/langchain-ai/langserve/tree/main/examples/file_processing/client.ipynb)                               | 
+| **LCEL Example** Example that uses LCEL to manipulate a dictionary input.                                                                                                                                                                                          | [server](https://github.com/langchain-ai/langserve/tree/main/examples/passthrough_dict/server.py), [client](https://github.com/langchain-ai/langserve/tree/main/examples/passthrough_dict/client.ipynb)                             |
+| **Auth** with `add_routes`: Simple authentication that can be applied across all endpoints associated with app. (Not useful on its own for implementing per user logic.)                                                                                           | [server](https://github.com/langchain-ai/langserve/tree/main/examples/auth/global_deps/server.py)                                                                                                                                   |
+| **Auth** with `add_routes`: Simple authentication mechanism based on path dependencies. (No useful on its own for implementing per user logic.)                                                                                                                    | [server](https://github.com/langchain-ai/langserve/tree/main/examples/auth/path_dependencies/server.py)                                                                                                                             |
+| **Auth** with `add_routes`: Implement per user logic and auth for endpoints that use per request config modifier. (**Note**: At the moment, does not integrate with OpenAPI docs.)                                                                                 | [server](https://github.com/langchain-ai/langserve/tree/main/examples/auth/per_req_config_modifier/server.py), [client](https://github.com/langchain-ai/langserve/tree/main/examples/auth/per_req_config_modifier/client.ipynb)     |
+| **Auth** with `APIHandler`: Implement per user logic and auth that shows how to search only within user owned documents.                                                                                                                                           | [server](https://github.com/langchain-ai/langserve/tree/main/examples/auth/api_handler/server.py), [client](https://github.com/langchain-ai/langserve/tree/main/examples/auth/api_handler/client.ipynb)                             |
+| **Widgets** Different widgets that can be used with playground (file upload and chat)                                                                                                                                                                              | [server](https://github.com/langchain-ai/langserve/tree/main/examples/widgets/chat/tuples/server.py)                                                                                                                                |
+| **Widgets** File upload widget used for LangServe playground.                                                                                                                                                                                                      | [server](https://github.com/langchain-ai/langserve/tree/main/examples/file_processing/server.py), [client](https://github.com/langchain-ai/langserve/tree/main/examples/file_processing/client.ipynb)                               |
 
 ## Sample Application
 
@@ -137,17 +167,17 @@ app = FastAPI(
 
 add_routes(
     app,
-    ChatOpenAI(),
+    ChatOpenAI(model="gpt-3.5-turbo-0125"),
     path="/openai",
 )
 
 add_routes(
     app,
-    ChatAnthropic(),
+    ChatAnthropic(model="claude-3-haiku-20240307"),
     path="/anthropic",
 )
 
-model = ChatAnthropic()
+model = ChatAnthropic(model="claude-3-haiku-20240307")
 prompt = ChatPromptTemplate.from_template("tell me a joke about {topic}")
 add_routes(
     app,
@@ -182,8 +212,8 @@ app.add_middleware(
 
 If you've deployed the server above, you can view the generated OpenAPI docs using:
 
-> ⚠️ If using pydantic v2, docs will not be generated for *invoke*, *batch*, *stream*,
-*stream_log*. See [Pydantic](#pydantic) section below for more details.
+> ⚠️ If using pydantic v2, docs will not be generated for _invoke_, _batch_, _stream_,
+> _stream_log_. See [Pydantic](#pydantic) section below for more details.
 
 ```sh
 curl localhost:8000/docs
@@ -243,10 +273,10 @@ In TypeScript (requires LangChain.js version 0.0.166 or later):
 import { RemoteRunnable } from "@langchain/core/runnables/remote";
 
 const chain = new RemoteRunnable({
-    url: `http://localhost:8000/joke/`,
+  url: `http://localhost:8000/joke/`,
 });
 const result = await chain.invoke({
-    topic: "cats",
+  topic: "cats",
 });
 ```
 
@@ -295,20 +325,20 @@ adds of these endpoints to the server:
 - `POST /my_runnable/stream_log` - invoke on a single input and stream the output,
   including output of intermediate steps as it's generated
 - `POST /my_runnable/astream_events` - invoke on a single input and stream events as they are generated,
- including from intermediate steps.
+  including from intermediate steps.
 - `GET /my_runnable/input_schema` - json schema for input to the runnable
 - `GET /my_runnable/output_schema` - json schema for output of the runnable
 - `GET /my_runnable/config_schema` - json schema for config of the runnable
 
 These endpoints match
-the [LangChain Expression Language interface](https://python.langchain.com/docs/expression_language/interface) --
+the [LangChain Expression Language interface](https://python.langchain.com/v0.2/docs/how_to/lcel_cheatsheet/) --
 please reference this documentation for more details.
 
 ## Playground
 
 You can find a playground page for your runnable at `/my_runnable/playground/`. This
 exposes a simple UI
-to [configure](https://python.langchain.com/docs/expression_language/how_to/configure)
+to [configure](https://python.langchain.com/v0.2/docs/how_to/configure/)
 and invoke your runnable with streaming output and intermediate steps.
 
 <p align="center">
@@ -332,12 +362,18 @@ runnable and share a link with the configuration:
 
 ## Chat playground
 
-LangServe also makes a chat-focused playground available at `/my_runnable/chat_playground/`.
+LangServe also supports a chat-focused playground that opt into and use under `/my_runnable/playground/`.
 Unlike the general playground, only certain types of runnables are supported - the runnable's input schema must
-be a `dict` with a single key, and that key's value must be a list of chat messages. The runnable
-can return either an `AIMessage` or a string.
+be a `dict` with either:
 
-Here's an example route:
+- a single key, and that key's value must be a list of chat messages.
+- two keys, one whose value is a list of messages, and the other representing the most recent message.
+
+We recommend you use the first format.
+
+The runnable must also return either an `AIMessage` or a string.
+
+To enable it, you must set `playground_type="chat",` when adding your route. Here's an example:
 
 ```python
 # Declare a chain
@@ -365,6 +401,7 @@ add_routes(
     chain.with_types(input_type=InputChat),
     enable_feedback_endpoint=True,
     enable_public_trace_link_endpoint=True,
+    playground_type="chat",
 )
 ```
 
@@ -390,7 +427,7 @@ for demos or testing.
 ## Legacy Chains
 
 LangServe works with both Runnables (constructed
-via [LangChain Expression Language](https://python.langchain.com/docs/expression_language/))
+via [LangChain Expression Language](https://python.langchain.com/v0.2/docs/how_to/#langchain-expression-language-lcel))
 and legacy chains (inheriting from `Chain`).
 However, some of the input schemas for legacy chains may be incomplete/incorrect,
 leading to errors.
@@ -429,27 +466,6 @@ You can deploy to GCP Cloud Run using the following command:
 gcloud run deploy [your-service-name] --source . --port 8001 --allow-unauthenticated --region us-central1 --set-env-vars=OPENAI_API_KEY=your_key
 ```
 
-### Deploy using Infrastructure as Code
-
-#### Pulumi
-
-You can deploy your LangServe server with [Pulumi](https://www.pulumi.com/) using your preferred general purpose language. Below are some quickstart 
-examples for deploying LangServe to different cloud providers.
-
-These examples are a good starting point for your own infrastructure as code (IaC) projects. You can easily modify them to suit your needs.
-
-
-| Cloud | Language   | Repository                                          | Quickstart                                                                                                                                  |
-|-------|------------|-----------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------|
-| AWS   | dotnet     | https://github.com/pulumi/examples/aws-cs-langserve | [![Deploy](https://get.pulumi.com/new/button.svg)](https://app.pulumi.com/new?template=https://github.com/pulumi/examples/aws-cs-langserve) |
-| AWS   | golang     | https://github.com/pulumi/examples/aws-go-langserve | [![Deploy](https://get.pulumi.com/new/button.svg)](https://app.pulumi.com/new?template=https://github.com/pulumi/examples/aws-go-langserve) |
-| AWS   | python     | https://github.com/pulumi/examples/aws-py-langserve | [![Deploy](https://get.pulumi.com/new/button.svg)](https://app.pulumi.com/new?template=https://github.com/pulumi/examples/aws-py-langserve) |
-| AWS   | typescript | https://github.com/pulumi/examples/aws-ts-langserve | [![Deploy](https://get.pulumi.com/new/button.svg)](https://app.pulumi.com/new?template=https://github.com/pulumi/examples/aws-ts-langserve) |
-| AWS   | javascript | https://github.com/pulumi/examples/aws-js-langserve | [![Deploy](https://get.pulumi.com/new/button.svg)](https://app.pulumi.com/new?template=https://github.com/pulumi/examples/aws-js-langserve) |
-
-
-
-
 ### Community Contributed
 
 #### Deploy to Railway
@@ -480,7 +496,7 @@ and [security](https://fastapi.tiangolo.com/tutorial/security/).
 
 The below examples show how to wire up authentication logic LangServe endpoints using FastAPI primitives.
 
-You are responsible for providing the actual authentication logic, the users table etc. 
+You are responsible for providing the actual authentication logic, the users table etc.
 
 If you're not sure what you're doing, you could try using an existing solution [Auth0](https://auth0.com/).
 
@@ -489,11 +505,11 @@ If you're not sure what you're doing, you could try using an existing solution [
 If you're using `add_routes`, see
 examples [here](https://github.com/langchain-ai/langserve/tree/main/examples/auth).
 
-| Description                                                                                                                                                                                                                                                        | Links                                                                                                                                                                                                                                |
-|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| **Auth** with `add_routes`: Simple authentication that can be applied across all endpoints associated with app. (Not useful on its own for implementing per user logic.)                                                                                           | [server](https://github.com/langchain-ai/langserve/tree/main/examples/auth/global_deps/server.py)                                                                                                                                    | 
-| **Auth** with `add_routes`: Simple authentication mechanism based on path dependencies. (No useful on its own for implementing per user logic.)                                                                                                                    | [server](https://github.com/langchain-ai/langserve/tree/main/examples/auth/path_dependencies/server.py)                                                                                                                              | 
-| **Auth** with `add_routes`: Implement per user logic and auth for endpoints that use per request config modifier. (**Note**: At the moment, does not integrate with OpenAPI docs.)                                                                                 | [server](https://github.com/langchain-ai/langserve/tree/main/examples/auth/per_req_config_modifier/server.py), [client](https://github.com/langchain-ai/langserve/tree/main/examples/auth/per_req_config_modifier/client.ipynb)     | 
+| Description                                                                                                                                                                        | Links                                                                                                                                                                                                                           |
+| :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| **Auth** with `add_routes`: Simple authentication that can be applied across all endpoints associated with app. (Not useful on its own for implementing per user logic.)           | [server](https://github.com/langchain-ai/langserve/tree/main/examples/auth/global_deps/server.py)                                                                                                                               |
+| **Auth** with `add_routes`: Simple authentication mechanism based on path dependencies. (No useful on its own for implementing per user logic.)                                    | [server](https://github.com/langchain-ai/langserve/tree/main/examples/auth/path_dependencies/server.py)                                                                                                                         |
+| **Auth** with `add_routes`: Implement per user logic and auth for endpoints that use per request config modifier. (**Note**: At the moment, does not integrate with OpenAPI docs.) | [server](https://github.com/langchain-ai/langserve/tree/main/examples/auth/per_req_config_modifier/server.py), [client](https://github.com/langchain-ai/langserve/tree/main/examples/auth/per_req_config_modifier/client.ipynb) |
 
 Alternatively, you can use FastAPI's [middleware](https://fastapi.tiangolo.com/tutorial/middleware/).
 
@@ -513,10 +529,10 @@ authorization purposes.
 
 If you feel comfortable with FastAPI and python, you can use LangServe's [APIHandler](https://github.com/langchain-ai/langserve/blob/main/examples/api_handler_examples/server.py).
 
-| Description                                                                                                                                                                                                                                                        | Links                                                                                                                                                                                                                                |
-|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| **Auth** with `APIHandler`: Implement per user logic and auth that shows how to search only within user owned documents.                                                                                                                                           | [server](https://github.com/langchain-ai/langserve/tree/main/examples/auth/api_handler/server.py), [client](https://github.com/langchain-ai/langserve/tree/main/examples/auth/api_handler/client.ipynb)                             | 
-| **APIHandler** Shows how to use `APIHandler` instead of `add_routes`. This provides more flexibility for developers to define endpoints. Works well with all FastAPI patterns, but takes a bit more effort.                                                        | [server](https://github.com/langchain-ai/langserve/tree/main/examples/api_handler_examples/server.py), [client](https://github.com/langchain-ai/langserve/tree/main/examples/api_handler_examples/client.ipynb)                     |
+| Description                                                                                                                                                                                                 | Links                                                                                                                                                                                                           |
+| :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| **Auth** with `APIHandler`: Implement per user logic and auth that shows how to search only within user owned documents.                                                                                    | [server](https://github.com/langchain-ai/langserve/tree/main/examples/auth/api_handler/server.py), [client](https://github.com/langchain-ai/langserve/tree/main/examples/auth/api_handler/client.ipynb)         |
+| **APIHandler** Shows how to use `APIHandler` instead of `add_routes`. This provides more flexibility for developers to define endpoints. Works well with all FastAPI patterns, but takes a bit more effort. | [server](https://github.com/langchain-ai/langserve/tree/main/examples/api_handler_examples/server.py), [client](https://github.com/langchain-ai/langserve/tree/main/examples/api_handler_examples/client.ipynb) |
 
 It's a bit more work, but gives you complete control over the endpoint definitions, so
 you can do whatever custom logic you need for auth.
@@ -585,8 +601,8 @@ add_routes(app, runnable)
 Inherit from `CustomUserType` if you want the data to de-serialize into a
 pydantic model rather than the equivalent dict representation.
 
-At the moment, this type only works *server* side and is used
-to specify desired *decoding* behavior. If inheriting from this type
+At the moment, this type only works _server_ side and is used
+to specify desired _decoding_ behavior. If inheriting from this type
 the server will keep the decoded type as a pydantic model instead
 of converting it into a dict.
 
@@ -625,10 +641,10 @@ The playground allows you to define custom widgets for your runnable from the ba
 
 Here are a few examples:
 
-| Description                                                                                                                                                                                                                                                        | Links                                                                                                                                                                                                                                |
-|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| **Widgets** Different widgets that can be used with playground (file upload and chat)                                                                                                                                                                              | [server](https://github.com/langchain-ai/langserve/tree/main/examples/widgets/chat/tuples/server.py), [client](https://github.com/langchain-ai/langserve/tree/main/examples/widgets/client.ipynb)                                               | 
-| **Widgets** File upload widget used for LangServe playground.                                                                                                                                                                                                      | [server](https://github.com/langchain-ai/langserve/tree/main/examples/file_processing/server.py), [client](https://github.com/langchain-ai/langserve/tree/main/examples/file_processing/client.ipynb)                               | 
+| Description                                                                           | Links                                                                                                                                                                                                 |
+| :------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| **Widgets** Different widgets that can be used with playground (file upload and chat) | [server](https://github.com/langchain-ai/langserve/tree/main/examples/widgets/chat/tuples/server.py), [client](https://github.com/langchain-ai/langserve/tree/main/examples/widgets/client.ipynb)     |
+| **Widgets** File upload widget used for LangServe playground.                         | [server](https://github.com/langchain-ai/langserve/tree/main/examples/file_processing/server.py), [client](https://github.com/langchain-ai/langserve/tree/main/examples/file_processing/client.ipynb) |
 
 #### Schema
 
@@ -644,8 +660,8 @@ type NameSpacedPath = { title: string; path: JsonPath }; // Using title to mimic
 type OneOfPath = { oneOf: JsonPath[] };
 
 type Widget = {
-    type: string // Some well known type (e.g., base64file, chat etc.)
-    [key: string]: JsonPath | NameSpacedPath | OneOfPath;
+  type: string; // Some well known type (e.g., base64file, chat etc.)
+  [key: string]: JsonPath | NameSpacedPath | OneOfPath;
 };
 ```
 
@@ -703,9 +719,9 @@ at the [widget example](https://github.com/langchain-ai/langserve/tree/main/exam
 
 To define a chat widget, make sure that you pass "type": "chat".
 
-* "input" is JSONPath to the field in the *Request* that has the new input message.
-* "output" is JSONPath to the field in the *Response* that has new output message(s).
-* Don't specify these fields if the entire input or output should be used as they are (
+- "input" is JSONPath to the field in the _Request_ that has the new input message.
+- "output" is JSONPath to the field in the _Response_ that has new output message(s).
+- Don't specify these fields if the entire input or output should be used as they are (
   e.g., if the output is a list of chat messages.)
 
 Here's a snippet:
@@ -745,6 +761,7 @@ add_routes(
 ```
 
 Example widget:
+
 <p align="center">
 <img src="https://github.com/langchain-ai/langserve/assets/3205522/a71ff37b-a6a9-4857-a376-cf27c41d3ca4" width="50%"/>
 </p>

SECURITY.md

@@ -1,6 +1,61 @@
 # Security Policy
 
-## Reporting a Vulnerability
+## Reporting OSS Vulnerabilities
 
-Please report security vulnerabilities by email to `security@langchain.dev`.
-This email is an alias to a subset of our maintainers, and will ensure the issue is promptly triaged and acted upon as needed.
+LangChain is partnered with [huntr by Protect AI](https://huntr.com/) to provide 
+a bounty program for our open source projects. 
+
+Please report security vulnerabilities associated with the LangChain 
+open source projects by visiting the following link:
+
+[https://huntr.com/bounties/disclose/](https://huntr.com/bounties/disclose/?target=https%3A%2F%2Fgithub.com%2Flangchain-ai%2Flangchain&validSearch=true)
+
+Before reporting a vulnerability, please review:
+
+1) In-Scope Targets and Out-of-Scope Targets below.
+2) The [langchain-ai/langchain](https://python.langchain.com/docs/contributing/repo_structure) monorepo structure.
+3) LangChain [security guidelines](https://python.langchain.com/docs/security) to
+   understand what we consider to be a security vulnerability vs. developer
+   responsibility.
+
+### In-Scope Targets
+
+The following packages and repositories are eligible for bug bounties:
+
+- langchain-core
+- langchain (see exceptions)
+- langchain-community (see exceptions)
+- langgraph
+- langserve
+
+### Out of Scope Targets
+
+All out of scope targets defined by huntr as well as:
+
+- **langchain-experimental**: This repository is for experimental code and is not
+  eligible for bug bounties, bug reports to it will be marked as interesting or waste of
+  time and published with no bounty attached.
+- **tools**: Tools in either langchain or langchain-community are not eligible for bug
+  bounties. This includes the following directories
+  - langchain/tools
+  - langchain-community/tools
+  - Please review our [security guidelines](https://python.langchain.com/docs/security)
+    for more details, but generally tools interact with the real world. Developers are
+    expected to understand the security implications of their code and are responsible
+    for the security of their tools.
+- Code documented with security notices. This will be decided done on a case by
+  case basis, but likely will not be eligible for a bounty as the code is already
+  documented with guidelines for developers that should be followed for making their
+  application secure.
+- Any LangSmith related repositories or APIs see below.
+
+## Reporting LangSmith Vulnerabilities
+
+Please report security vulnerabilities associated with LangSmith by email to `security@langchain.dev`.
+
+- LangSmith site: https://smith.langchain.com
+- SDK client: https://github.com/langchain-ai/langsmith-sdk
+
+### Other Security Concerns
+
+For any other security concerns, please contact us at `security@langchain.dev`.

examples/agent/server.py

@@ -20,15 +20,15 @@ Relevant LangChain documentation:
 from typing import Any
 
 from fastapi import FastAPI
-from langchain.agents import AgentExecutor, tool
+from langchain.agents import AgentExecutor
 from langchain.agents.format_scratchpad import format_to_openai_functions
 from langchain.agents.output_parsers import OpenAIFunctionsAgentOutputParser
-from langchain.chat_models import ChatOpenAI
-from langchain.embeddings import OpenAIEmbeddings
-from langchain.prompts import ChatPromptTemplate, MessagesPlaceholder
 from langchain.pydantic_v1 import BaseModel
-from langchain.tools.render import format_tool_to_openai_function
-from langchain.vectorstores import FAISS
+from langchain_community.vectorstores import FAISS
+from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
+from langchain_core.tools import tool
+from langchain_core.utils.function_calling import format_tool_to_openai_function
+from langchain_openai import ChatOpenAI, OpenAIEmbeddings
 
 from langserve import add_routes
 

examples/agent_custom_streaming/server.py

@@ -47,15 +47,15 @@ Relevant LangChain documentation:
 from typing import Any, AsyncIterator, List, Literal
 
 from fastapi import FastAPI
-from langchain.agents import AgentExecutor, tool
+from langchain.agents import AgentExecutor
 from langchain.agents.format_scratchpad.openai_tools import (
     format_to_openai_tool_messages,
 )
 from langchain.agents.output_parsers.openai_tools import OpenAIToolsAgentOutputParser
-from langchain.prompts import MessagesPlaceholder
-from langchain_community.tools.convert_to_openai import format_tool_to_openai_tool
-from langchain_core.prompts import ChatPromptTemplate
+from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
 from langchain_core.runnables import RunnableLambda
+from langchain_core.tools import tool
+from langchain_core.utils.function_calling import format_tool_to_openai_tool
 from langchain_openai import ChatOpenAI
 
 from langserve import add_routes

examples/agent_with_history/server.py

@@ -26,15 +26,15 @@ Relevant LangChain documentation:
 from typing import Any, List, Union
 
 from fastapi import FastAPI
-from langchain.agents import AgentExecutor, tool
+from langchain.agents import AgentExecutor
 from langchain.agents.format_scratchpad.openai_tools import (
     format_to_openai_tool_messages,
 )
 from langchain.agents.output_parsers.openai_tools import OpenAIToolsAgentOutputParser
-from langchain.prompts import MessagesPlaceholder
-from langchain_community.tools.convert_to_openai import format_tool_to_openai_tool
 from langchain_core.messages import AIMessage, FunctionMessage, HumanMessage
-from langchain_core.prompts import ChatPromptTemplate
+from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
+from langchain_core.tools import tool
+from langchain_core.utils.function_calling import format_tool_to_openai_tool
 from langchain_openai import ChatOpenAI
 
 from langserve import add_routes

examples/auth/api_handler/server.py

@@ -35,7 +35,6 @@ from typing import Any, List, Optional, Union
 
 from fastapi import Depends, FastAPI, HTTPException, Request, Response, status
 from fastapi.security import OAuth2PasswordBearer, OAuth2PasswordRequestForm
-from langchain_community.embeddings.openai import OpenAIEmbeddings
 from langchain_community.vectorstores.chroma import Chroma
 from langchain_core.documents import Document
 from langchain_core.runnables import (
@@ -44,6 +43,7 @@ from langchain_core.runnables import (
     RunnableSerializable,
 )
 from langchain_core.vectorstores import VectorStore
+from langchain_openai import OpenAIEmbeddings
 from typing_extensions import Annotated
 
 from langserve import APIHandler

examples/auth/per_req_config_modifier/server.py

@@ -36,7 +36,6 @@ from typing import Any, Dict, List, Optional, Union
 
 from fastapi import Depends, FastAPI, HTTPException, Request, status
 from fastapi.security import OAuth2PasswordBearer, OAuth2PasswordRequestForm
-from langchain_community.embeddings.openai import OpenAIEmbeddings
 from langchain_community.vectorstores.chroma import Chroma
 from langchain_core.documents import Document
 from langchain_core.runnables import (
@@ -45,6 +44,7 @@ from langchain_core.runnables import (
     RunnableSerializable,
 )
 from langchain_core.vectorstores import VectorStore
+from langchain_openai import OpenAIEmbeddings
 from typing_extensions import Annotated
 
 from langserve import add_routes

examples/chat_playground/legacy_input/server.py

@@ -0,0 +1,56 @@
+#!/usr/bin/env python
+"""Example of a simple chatbot that just passes current conversation
+state back and forth between server and client.
+"""
+from typing import List, Union
+
+from fastapi import FastAPI
+from langchain_anthropic import ChatAnthropic
+from langchain_core.messages import AIMessage, HumanMessage, SystemMessage
+from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
+
+from langserve import add_routes
+from langserve.pydantic_v1 import BaseModel, Field
+
+app = FastAPI(
+    title="LangChain Server",
+    version="1.0",
+    description="Spin up a simple api server using Langchain's Runnable interfaces",
+)
+
+
+# Declare a chain
+prompt = ChatPromptTemplate.from_messages(
+    [
+        ("system", "You are a helpful, professional assistant named Cob."),
+        MessagesPlaceholder(variable_name="messages"),
+        ("human", "{input}"),
+    ]
+)
+
+chain = prompt | ChatAnthropic(model="claude-2")
+
+
+class InputChat(BaseModel):
+    """Input for the chat endpoint."""
+
+    messages: List[Union[HumanMessage, AIMessage, SystemMessage]] = Field(
+        ...,
+        description="The chat messages representing the current conversation.",
+    )
+
+    input: str
+
+
+add_routes(
+    app,
+    chain.with_types(input_type=InputChat),
+    enable_feedback_endpoint=True,
+    enable_public_trace_link_endpoint=True,
+    playground_type="chat",
+)
+
+if __name__ == "__main__":
+    import uvicorn
+
+    uvicorn.run(app, host="localhost", port=8000)

examples/chat_playground/server.py

@@ -5,8 +5,7 @@ state back and forth between server and client.
 from typing import List, Union
 
 from fastapi import FastAPI
-from fastapi.middleware.cors import CORSMiddleware
-from langchain.chat_models import ChatAnthropic
+from langchain_anthropic.chat_models import ChatAnthropic
 from langchain_core.messages import AIMessage, HumanMessage, SystemMessage
 from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
 
@@ -20,17 +19,6 @@ app = FastAPI(
 )
 
 
-# Set all CORS enabled origins
-app.add_middleware(
-    CORSMiddleware,
-    allow_origins=["*"],
-    allow_credentials=True,
-    allow_methods=["*"],
-    allow_headers=["*"],
-    expose_headers=["*"],
-)
-
-
 # Declare a chain
 prompt = ChatPromptTemplate.from_messages(
     [
@@ -39,7 +27,7 @@ prompt = ChatPromptTemplate.from_messages(
     ]
 )
 
-chain = prompt | ChatAnthropic(model="claude-2")
+chain = prompt | ChatAnthropic(model_name="claude-3-sonnet-20240229")
 
 
 class InputChat(BaseModel):
@@ -56,6 +44,7 @@ add_routes(
     chain.with_types(input_type=InputChat),
     enable_feedback_endpoint=True,
     enable_public_trace_link_endpoint=True,
+    playground_type="chat",
 )
 
 if __name__ == "__main__":

examples/chat_with_persistence/server.py

@@ -13,8 +13,8 @@ from pathlib import Path
 from typing import Callable, Union
 
 from fastapi import FastAPI, HTTPException
-from langchain.chat_models import ChatAnthropic
-from langchain.memory import FileChatMessageHistory
+from langchain_anthropic import ChatAnthropic
+from langchain_community.chat_message_histories import FileChatMessageHistory
 from langchain_core.chat_history import BaseChatMessageHistory
 from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
 from langchain_core.runnables.history import RunnableWithMessageHistory

examples/chat_with_persistence_and_user/server.py

@@ -13,13 +13,13 @@ from pathlib import Path
 from typing import Any, Callable, Dict, Union
 
 from fastapi import FastAPI, HTTPException, Request
-from langchain.chat_models import ChatOpenAI
-from langchain.memory import FileChatMessageHistory
-from langchain.schema.runnable.utils import ConfigurableFieldSpec
+from langchain_community.chat_message_histories import FileChatMessageHistory
 from langchain_core import __version__
 from langchain_core.chat_history import BaseChatMessageHistory
 from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
+from langchain_core.runnables import ConfigurableFieldSpec
 from langchain_core.runnables.history import RunnableWithMessageHistory
+from langchain_openai import ChatOpenAI
 from typing_extensions import TypedDict
 
 from langserve import add_routes

examples/configurable_agent_executor/server.py

@@ -17,15 +17,12 @@ on the LLM and use the stream_log endpoint rather than stream endpoint.
 from typing import Any, AsyncIterator, Dict, List, Optional, cast
 
 from fastapi import FastAPI
-from langchain.agents import AgentExecutor, tool
+from langchain.agents import AgentExecutor
 from langchain.agents.format_scratchpad import format_to_openai_functions
 from langchain.agents.output_parsers import OpenAIFunctionsAgentOutputParser
-from langchain.chat_models import ChatOpenAI
-from langchain.embeddings import OpenAIEmbeddings
-from langchain.prompts import ChatPromptTemplate, MessagesPlaceholder
 from langchain.pydantic_v1 import BaseModel
-from langchain.tools.render import format_tool_to_openai_function
-from langchain.vectorstores import FAISS
+from langchain_community.vectorstores import FAISS
+from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
 from langchain_core.runnables import (
     ConfigurableField,
     ConfigurableFieldSpec,
@@ -33,6 +30,9 @@ from langchain_core.runnables import (
     RunnableConfig,
 )
 from langchain_core.runnables.utils import Input, Output
+from langchain_core.tools import tool
+from langchain_core.utils.function_calling import format_tool_to_openai_function
+from langchain_openai import ChatOpenAI, OpenAIEmbeddings
 
 from langserve import add_routes
 

examples/configurable_chain/server.py

@@ -10,10 +10,10 @@ from typing import Any, Dict
 
 from fastapi import FastAPI, HTTPException, Request
 from fastapi.middleware.cors import CORSMiddleware
-from langchain.chat_models import ChatOpenAI
-from langchain.prompts import PromptTemplate
-from langchain.schema.output_parser import StrOutputParser
-from langchain.schema.runnable import ConfigurableField
+from langchain_core.output_parsers import StrOutputParser
+from langchain_core.prompts import PromptTemplate
+from langchain_core.runnables import ConfigurableField
+from langchain_openai import ChatOpenAI
 
 from langserve import add_routes
 

examples/configurable_retrieval/server.py

@@ -3,17 +3,18 @@
 from typing import Any, Iterable, List, Optional, Type
 
 from fastapi import FastAPI
-from langchain.embeddings import OpenAIEmbeddings
-from langchain.schema import Document
-from langchain.schema.embeddings import Embeddings
-from langchain.schema.retriever import BaseRetriever
-from langchain.schema.runnable import (
+from langchain.schema.vectorstore import VST
+from langchain_community.vectorstores import FAISS
+from langchain_core.documents import Document
+from langchain_core.embeddings import Embeddings
+from langchain_core.retrievers import BaseRetriever
+from langchain_core.runnables import (
     ConfigurableFieldSingleOption,
     RunnableConfig,
     RunnableSerializable,
 )
-from langchain.schema.vectorstore import VST
-from langchain.vectorstores import FAISS, VectorStore
+from langchain_core.vectorstores import VectorStore
+from langchain_openai import OpenAIEmbeddings
 
 from langserve import add_routes
 from langserve.pydantic_v1 import BaseModel, Field

examples/conversational_retrieval_chain/server.py

@@ -13,14 +13,11 @@ from operator import itemgetter
 from typing import List, Tuple
 
 from fastapi import FastAPI
-from langchain.chat_models import ChatOpenAI
-from langchain.embeddings import OpenAIEmbeddings
-from langchain.prompts import ChatPromptTemplate
-from langchain.prompts.prompt import PromptTemplate
-from langchain.schema import format_document
-from langchain.schema.output_parser import StrOutputParser
-from langchain.schema.runnable import RunnableMap, RunnablePassthrough
-from langchain.vectorstores import FAISS
+from langchain_community.vectorstores import FAISS
+from langchain_core.output_parsers import StrOutputParser
+from langchain_core.prompts import ChatPromptTemplate, PromptTemplate, format_document
+from langchain_core.runnables import RunnableMap, RunnablePassthrough
+from langchain_openai import ChatOpenAI, OpenAIEmbeddings
 
 from langserve import add_routes
 from langserve.pydantic_v1 import BaseModel, Field

examples/file_processing/server.py

@@ -15,10 +15,10 @@ allowing one to upload a binary file using the langserve playground UI.
 import base64
 
 from fastapi import FastAPI
-from langchain.document_loaders.blob_loaders import Blob
-from langchain.document_loaders.parsers.pdf import PDFMinerParser
 from langchain.pydantic_v1 import Field
-from langchain.schema.runnable import RunnableLambda
+from langchain_community.document_loaders.parsers.pdf import PDFMinerParser
+from langchain_core.document_loaders import Blob
+from langchain_core.runnables import RunnableLambda
 
 from langserve import CustomUserType, add_routes
 

examples/llm/server.py

@@ -2,7 +2,8 @@
 """Example LangChain server exposes multiple runnables (LLMs in this case)."""
 
 from fastapi import FastAPI
-from langchain.chat_models import ChatAnthropic, ChatOpenAI
+from langchain_anthropic import ChatAnthropic
+from langchain_openai import ChatOpenAI
 
 from langserve import add_routes
 
@@ -14,12 +15,12 @@ app = FastAPI(
 
 add_routes(
     app,
-    ChatOpenAI(),
+    ChatOpenAI(model="gpt-3.5-turbo-0125"),
     path="/openai",
 )
 add_routes(
     app,
-    ChatAnthropic(),
+    ChatAnthropic(model="claude-3-haiku-20240307"),
     path="/anthropic",
 )
 

examples/passthrough_dict/server.py

@@ -4,9 +4,9 @@
 from typing import Any, Callable, Dict, List, Optional, TypedDict
 
 from fastapi import FastAPI
-from langchain.chat_models import ChatOpenAI
-from langchain.prompts import ChatPromptTemplate
-from langchain.schema.runnable import RunnableMap, RunnablePassthrough
+from langchain_core.prompts import ChatPromptTemplate
+from langchain_core.runnables import RunnableParallel, RunnablePassthrough
+from langchain_openai import ChatOpenAI
 
 from langserve import add_routes
 
@@ -43,7 +43,7 @@ model = ChatOpenAI()
 
 underlying_chain = prompt | model
 
-wrapped_chain = RunnableMap(
+wrapped_chain = RunnableParallel(
     {
         "output": _create_projection(exclude_keys=["info"]) | underlying_chain,
         "info": _create_projection(include_keys=["info"]),

examples/retrieval/server.py

@@ -1,8 +1,8 @@
 #!/usr/bin/env python
 """Example LangChain server exposes a retriever."""
 from fastapi import FastAPI
-from langchain.embeddings import OpenAIEmbeddings
-from langchain.vectorstores import FAISS
+from langchain_community.vectorstores import FAISS
+from langchain_openai import OpenAIEmbeddings
 
 from langserve import add_routes
 

examples/router/server.py

@@ -9,7 +9,8 @@ See more documentation at:
 https://fastapi.tiangolo.com/tutorial/bigger-applications/
 """
 from fastapi import APIRouter, FastAPI
-from langchain.chat_models import ChatAnthropic, ChatOpenAI
+from langchain_anthropic import ChatAnthropic
+from langchain_openai import ChatOpenAI
 
 from langserve import add_routes
 
@@ -20,13 +21,13 @@ router = APIRouter(prefix="/models")
 # Invocations to this router will appear in trace logs as /models/openai
 add_routes(
     router,
-    ChatOpenAI(),
+    ChatOpenAI(model="gpt-3.5-turbo-0125"),
     path="/openai",
 )
 # Invocations to this router will appear in trace logs as /models/anthropic
 add_routes(
     router,
-    ChatAnthropic(),
+    ChatAnthropic(model="claude-3-haiku-20240307"),
     path="/anthropic",
 )
 

examples/widgets/chat/message_list/server.py

@@ -6,7 +6,7 @@ from typing import List, Union
 
 from fastapi import FastAPI
 from fastapi.middleware.cors import CORSMiddleware
-from langchain.chat_models import ChatAnthropic
+from langchain_anthropic import ChatAnthropic
 from langchain_core.messages import AIMessage, HumanMessage, SystemMessage
 from langchain_core.output_parsers import StrOutputParser
 from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder

examples/widgets/chat/tuples/server.py

@@ -6,18 +6,17 @@ from typing import Any, Dict, List, Tuple
 
 from fastapi import FastAPI
 from fastapi.middleware.cors import CORSMiddleware
-from langchain.chat_models.openai import ChatOpenAI
-from langchain.document_loaders.blob_loaders import Blob
-from langchain.document_loaders.parsers.pdf import PDFMinerParser
 from langchain.pydantic_v1 import BaseModel, Field
-from langchain.schema.messages import (
+from langchain_community.document_loaders.parsers.pdf import PDFMinerParser
+from langchain_core.document_loaders import Blob
+from langchain_core.messages import (
     AIMessage,
     BaseMessage,
     FunctionMessage,
     HumanMessage,
 )
-from langchain.schema.runnable import RunnableLambda
-from langchain_core.runnables import RunnableParallel
+from langchain_core.runnables import RunnableLambda, RunnableParallel
+from langchain_openai import ChatOpenAI
 
 from langserve import CustomUserType
 from langserve.server import add_routes

langserve/api_handler.py

@@ -1,9 +1,11 @@
+import asyncio
 import contextlib
 import importlib
 import inspect
 import json
 import os
 import re
+import uuid
 from inspect import isclass
 from typing import (
     Any,
@@ -12,6 +14,7 @@ from typing import (
     Callable,
     Dict,
     Generator,
+    List,
     Literal,
     Mapping,
     Optional,
@@ -24,18 +27,24 @@ from typing import (
 from fastapi import HTTPException
 from fastapi.encoders import jsonable_encoder
 from fastapi.exceptions import RequestValidationError
-from langchain.callbacks.base import AsyncCallbackHandler
-from langchain.callbacks.tracers.log_stream import RunLogPatch
-from langchain.load.serializable import Serializable
-from langchain.schema.runnable import Runnable, RunnableConfig
-from langchain.schema.runnable.config import get_config_list, merge_configs
+from langchain_core._api.beta_decorator import warn_beta
+from langchain_core.callbacks.base import AsyncCallbackHandler
+from langchain_core.load.serializable import Serializable
+from langchain_core.runnables import Runnable, RunnableConfig
+from langchain_core.runnables.config import (
+    get_config_list,
+    merge_configs,
+    run_in_executor,
+)
+from langchain_core.tracers import RunLogPatch
 from langsmith import client as ls_client
+from langsmith.schemas import FeedbackIngestToken
 from langsmith.utils import tracing_is_enabled
 from starlette.requests import Request
 from starlette.responses import JSONResponse, Response
+from typing_extensions import TypedDict
 
 from langserve.callbacks import AsyncEventAggregatorCallback, CallbackEventDict
-from langserve.chat_playground import serve_chat_playground
 from langserve.lzstring import LZString
 from langserve.playground import serve_playground
 from langserve.pydantic_v1 import BaseModel, Field, ValidationError, create_model
@@ -44,9 +53,11 @@ from langserve.schema import (
     CustomUserType,
     Feedback,
     FeedbackCreateRequest,
+    FeedbackCreateRequestTokenBased,
+    FeedbackToken,
+    InvokeResponseMetadata,
     PublicTraceLink,
     PublicTraceLinkCreateRequest,
-    SingletonResponseMetadata,
 )
 from langserve.serialization import WellKnownLCSerializer
 from langserve.validation import (
@@ -107,6 +118,39 @@ def _strip_internal_keys(metadata: Dict[str, Any]) -> Dict[str, Any]:
     return {k: v for k, v in metadata.items() if not k.startswith("__")}
 
 
+def _create_metadata_event(
+    run_id: Optional[uuid.UUID] = None,
+    feedback_key: Optional[str] = None,
+    feedback_ingest_token: Optional[FeedbackIngestToken] = None,
+) -> Dict[str, Any]:
+    """Create a metadata event with the given type and metadata."""
+    data = {
+        "run_id": str(run_id) if run_id else None,
+    }
+    if feedback_ingest_token:
+        if not feedback_key:
+            raise ValueError("Feedback key must be provided if feedback token is given")
+
+        if feedback_ingest_token.expires_at:
+            expires_at = feedback_ingest_token.expires_at.isoformat()
+        else:
+            expires_at = None
+
+        data["feedback_tokens"] = [
+            {
+                "key": feedback_key,
+                "token_url": feedback_ingest_token.url,
+                "expires_at": expires_at,
+            }
+        ]
+
+    metadata = {
+        "event": "metadata",
+        "data": json.dumps(data),
+    }
+    return metadata
+
+
 async def _unpack_request_config(
     *client_sent_configs: Union[BaseModel, Mapping, str],
     config_keys: Sequence[str],
@@ -219,12 +263,19 @@ def _update_config_with_defaults(
     # merge_configs is last-writer-wins, so we specifically pass in the
     # overridable configs first, then the user provided configs, then
     # finally the non-overridable configs
-    return merge_configs(
+    config = merge_configs(
         overridable_default_config,
         incoming_config,
         non_overridable_default_config,
     )
 
+    # run_id may have been set by user (and accepted by server) or
+    # it may have been by the user on the server request path.
+    # If it's not set, we'll generate a new one.
+    if "run_id" not in config or config["run_id"] is None:
+        config["run_id"] = str(uuid.uuid4())
+    return config
+
 
 def _unpack_input(validated_model: BaseModel) -> Any:
     """Unpack the decoded input from the validated model."""
@@ -315,7 +366,11 @@ def _add_namespace_to_model(namespace: str, model: Type[BaseModel]) -> Type[Base
         A new model with name prepended with the given namespace.
     """
     model_with_unique_name = _rename_pydantic_model(model, namespace)
-    model_with_unique_name.update_forward_refs()
+    if "run_id" in model_with_unique_name.__annotations__:
+        # Help resolve reference by providing namespace references
+        model_with_unique_name.update_forward_refs(uuid=uuid)
+    else:
+        model_with_unique_name.update_forward_refs()
     return model_with_unique_name
 
 
@@ -351,23 +406,6 @@ def _with_validation_error_translation() -> Generator[None, None, None]:
         raise RequestValidationError(e.errors(), body=e.model)
 
 
-def _get_base_run_id_as_str(
-    event_aggregator: AsyncEventAggregatorCallback,
-) -> Optional[str]:
-    """
-    Uses `event_aggregator` to determine the base run ID for a given run. Returns
-    the run_id as a string, or None if it does not exist.
-    """
-    # The first run in the callback_events list corresponds to the
-    # overall trace for request
-    if event_aggregator.callback_events and event_aggregator.callback_events[0].get(
-        "run_id"
-    ):
-        return str(event_aggregator.callback_events[0].get("run_id"))
-    else:
-        raise AssertionError("No run_id found for the given run")
-
-
 def _json_encode_response(model: BaseModel) -> JSONResponse:
     """Return a JSONResponse with the given content.
 
@@ -439,7 +477,23 @@ _MODEL_REGISTRY = {}
 _SEEN_NAMES = set()
 
 
+class PerKeyFeedbackConfig(TypedDict):
+    """Per feedback configuration.
+
+    Use to configure the feedback token.
+    """
+
+    key: str
+
+
 # PUBLIC API
+class TokenFeedbackConfig(TypedDict):
+    """Token feedback configuration.
+
+    This is used to configure the feedback tokens.
+    """
+
+    key_configs: List[PerKeyFeedbackConfig]
 
 
 class APIHandler:
@@ -465,9 +519,14 @@ class APIHandler:
         config_keys: Sequence[str] = ("configurable",),
         include_callback_events: bool = False,
         enable_feedback_endpoint: bool = False,
+        # token feedback config configures **token** based feedback which
+        # is different from the feedback endpoint.
+        # Read the documentation for more details
+        token_feedback_config: Optional[TokenFeedbackConfig] = None,
         enable_public_trace_link_endpoint: bool = False,
         per_req_config_modifier: Optional[PerRequestConfigModifier] = None,
         stream_log_name_allow_list: Optional[Sequence[str]] = None,
+        playground_type: Literal["default", "chat"] = "default",
     ) -> None:
         """Create an API handler for the given runnable.
 
@@ -506,6 +565,11 @@ class APIHandler:
                 to LangSmith. Disabled by default. If this flag is disabled or LangSmith
                 tracing is not enabled for the runnable, then 4xx errors will be thrown
                 when accessing the feedback endpoint
+                **Attention** this is distinct from `token_feedback_config`.
+            token_feedback_config: optional configuration for token based feedback.
+                **Attention** this is distinct from `enable_feedback_endpoint`.
+                When provided, feedback tokens will be included in the response
+                metadata that can be used to provide feedback on the run.
             enable_public_trace_link_endpoint:  Whether to enable an endpoint for
                 end-users to publicly view LangSmith traces of your chain runs.
                 WARNING: THIS WILL EXPOSE THE INTERNAL STATE OF YOUR RUN AND CHAIN AS
@@ -562,11 +626,27 @@ class APIHandler:
         self._enable_public_trace_link_endpoint = enable_public_trace_link_endpoint
         self._names_in_stream_allow_list = stream_log_name_allow_list
 
+        if token_feedback_config:
+            if len(token_feedback_config["key_configs"]) != 1:
+                raise NotImplementedError(
+                    "Only one key is supported for now for token feedback "
+                    "configuration. For example specify: "
+                    "{'key_configs': [{'key': 'correctness'}]}"
+                )
+
+            warn_beta(
+                message="Token feedback is in beta. This API may change in the future."
+            )
+
+        self._token_feedback_config = token_feedback_config
+        self._token_feedback_enabled = token_feedback_config is not None
+
         # Client is patched using mock.patch, if changing the names
         # remember to make relevant updates in the unit tests.
         self._langsmith_client = (
             ls_client.Client()
-            if tracing_is_enabled() and enable_feedback_endpoint
+            if tracing_is_enabled()
+            and (enable_feedback_endpoint or self._token_feedback_enabled)
             else None
         )
 
@@ -617,9 +697,12 @@ class APIHandler:
         )
         # Generate the response models
         self._InvokeResponse = create_invoke_response_model(
-            model_namespace, output_type_
+            model_namespace, output_type_, include_callback_events
         )
-        self._BatchResponse = create_batch_response_model(model_namespace, output_type_)
+        self._BatchResponse = create_batch_response_model(
+            model_namespace, output_type_, include_callback_events
+        )
+        self.playground_type = playground_type
 
     @property
     def InvokeRequest(self) -> Type[BaseModel]:
@@ -722,10 +805,33 @@ class APIHandler:
             endpoint="invoke",
             server_config=server_config,
         )
+        run_id = config["run_id"]
 
         event_aggregator = AsyncEventAggregatorCallback()
         _add_callbacks(config, [event_aggregator])
-        output = await self._runnable.ainvoke(input_, config=config)
+
+        invoke_coro = self._runnable.ainvoke(
+            input_,
+            config=config,
+        )
+
+        feedback_key: Optional[str]
+
+        # If there's feedback enabled, let's create a presigned feedback token
+        if self._token_feedback_enabled:
+            feedback_key = self._token_feedback_config["key_configs"][0]["key"]
+            feedback_coro = run_in_executor(
+                None,
+                self._langsmith_client.create_presigned_feedback_token,
+                run_id,
+                feedback_key,
+            )
+
+            output, feedback_token = await asyncio.gather(invoke_coro, feedback_coro)
+        else:
+            feedback_key = None
+            output = await invoke_coro
+            feedback_token = None
 
         if self._include_callback_events:
             callback_events = [
@@ -741,8 +847,17 @@ class APIHandler:
                 # Callbacks are scrubbed and exceptions are converted to
                 # serializable format before returned in the response.
                 callback_events=callback_events,
-                metadata=SingletonResponseMetadata(
-                    run_id=_get_base_run_id_as_str(event_aggregator)
+                metadata=InvokeResponseMetadata(
+                    run_id=run_id,
+                    feedback_tokens=[
+                        FeedbackToken(
+                            key=feedback_key,
+                            token_url=feedback_token.url,
+                            expires_at=feedback_token.expires_at.isoformat(),
+                        )
+                    ]
+                    if feedback_token
+                    else [],
                 ),
             ),
         )
@@ -838,7 +953,31 @@ class APIHandler:
                 )
             )
 
-        output = await self._runnable.abatch(inputs, config=final_configs)
+        run_ids = [config["run_id"] for config in final_configs]
+
+        batch_coro = self._runnable.abatch(inputs, config=final_configs)
+
+        feedback_key: Optional[str]
+
+        # If there's feedback enabled, let's create a presigned feedback token
+        if self._token_feedback_enabled:
+            feedback_key = self._token_feedback_config["key_configs"][0]["key"]
+            feedback_coros = [
+                run_in_executor(
+                    None,
+                    self._langsmith_client.create_presigned_feedback_token,
+                    run_id,
+                    feedback_key,
+                )
+                for run_id in run_ids
+            ]
+            response = await asyncio.gather(batch_coro, *feedback_coros)
+            output = response[0]
+            feedback_tokens = response[1:]
+        else:
+            feedback_key = None
+            output = await batch_coro
+            feedback_tokens = []
 
         if self._include_callback_events:
             callback_events = [
@@ -853,13 +992,35 @@ class APIHandler:
         else:
             callback_events = []
 
+        if feedback_tokens:
+            metadatas = [
+                InvokeResponseMetadata(
+                    run_id=run_id,
+                    feedback_tokens=[
+                        FeedbackToken(
+                            key=feedback_key,
+                            token_url=feedback_token.url,
+                            expires_at=feedback_token.expires_at.isoformat(),
+                        )
+                    ],
+                )
+                for run_id, feedback_token in zip(run_ids, feedback_tokens)
+            ]
+        else:
+            metadatas = [
+                InvokeResponseMetadata(run_id=run_id, feedback_tokens=[])
+                for run_id in run_ids
+            ]
+
         obj = self._BatchResponse(
             output=self._serializer.dumpd(output),
             callback_events=callback_events,
             metadata=BatchResponseMetadata(
-                run_ids=[_get_base_run_id_as_str(agg) for agg in aggregators]
+                run_ids=run_ids,
+                responses=metadatas,
             ),
         )
+
         return _json_encode_response(obj)
 
     async def stream(
@@ -920,8 +1081,7 @@ class APIHandler:
                 Originates from the client side. This config must be validated.
             server_config: optional server configuration that will be merged
         """
-        err_event = {}
-        validation_exception: Optional[BaseException] = None
+        run_id = None
         try:
             config, input_ = await self._get_config_and_input(
                 request,
@@ -929,36 +1089,30 @@ class APIHandler:
                 endpoint="stream",
                 server_config=server_config,
             )
-        except BaseException as e:
-            validation_exception = e
-            if isinstance(e, RequestValidationError):
-                err_event = {
-                    "event": "error",
-                    "data": json.dumps(
-                        {"status_code": 422, "message": repr(e.errors())}
-                    ),
-                }
-            else:
-                err_event = {
-                    "event": "error",
-                    # Do not expose the error message to the client since
-                    # the message may contain sensitive information.
-                    "data": json.dumps(
-                        {"status_code": 500, "message": "Internal Server Error"}
-                    ),
-                }
+            run_id = config["run_id"]
+        except BaseException:
+            # Exceptions will be properly translated by default FastAPI middleware
+            # to either 422 (on input validation) or 500 internal server errors.
+            raise
+
+        if self._token_feedback_enabled:
+            # Create task to create a presigned feedback token
+            feedback_key: Optional[str] = self._token_feedback_config["key_configs"][0][
+                "key"
+            ]
+            feedback_coro = run_in_executor(
+                None,
+                self._langsmith_client.create_presigned_feedback_token,
+                run_id,
+                feedback_key,
+            )
+            task: Optional[asyncio.Task] = asyncio.create_task(feedback_coro)
+        else:
+            feedback_key = None
+            task = None
 
         async def _stream() -> AsyncIterator[dict]:
             """Stream the output of the runnable."""
-            if validation_exception:
-                yield err_event
-                if isinstance(validation_exception, RequestValidationError):
-                    return
-                else:
-                    raise AssertionError(
-                        "Internal server error"
-                    ) from validation_exception
-
             try:
                 config_w_callbacks = config.copy()
                 event_aggregator = AsyncEventAggregatorCallback()
@@ -968,16 +1122,19 @@ class APIHandler:
                     input_,
                     config=config_w_callbacks,
                 ):
-                    if not has_sent_metadata and event_aggregator.callback_events:
-                        yield {
-                            "event": "metadata",
-                            "data": json.dumps(
-                                {
-                                    "run_id": _get_base_run_id_as_str(event_aggregator),
-                                }
-                            ),
-                        }
+                    # Send a metadata event as soon as possible
+                    if not has_sent_metadata:
+                        if task is not None and not task.done():
+                            continue
+                        if task is None:
+                            feedback_token = None
+                        else:
+                            feedback_token = task.result()
+
                         has_sent_metadata = True
+                        yield _create_metadata_event(
+                            run_id, feedback_key, feedback_token
+                        )
 
                     yield {
                         # EventSourceResponse expects a string for data
@@ -1013,8 +1170,6 @@ class APIHandler:
         View documentation for endpoint at the end of the file.
         It's attached to _stream_log_docs endpoint.
         """
-        err_event = {}
-        validation_exception: Optional[BaseException] = None
         try:
             config, input_ = await self._get_config_and_input(
                 request,
@@ -1022,56 +1177,39 @@ class APIHandler:
                 endpoint="stream_log",
                 server_config=server_config,
             )
-        except BaseException as e:
-            validation_exception = e
-            if isinstance(e, RequestValidationError):
-                err_event = {
-                    "event": "error",
-                    "data": json.dumps(
-                        {"status_code": 422, "message": repr(e.errors())}
-                    ),
-                }
-            else:
-                err_event = {
-                    "event": "error",
-                    # Do not expose the error message to the client since
-                    # the message may contain sensitive information.
-                    "data": json.dumps(
-                        {"status_code": 500, "message": "Internal Server Error"}
-                    ),
-                }
-
+            run_id = config["run_id"]
+        except BaseException:
+            # Exceptions will be properly translated by default FastAPI middleware
+            # to either 422 (on input validation) or 500 internal server errors.
+            raise
         try:
             body = await request.json()
             with _with_validation_error_translation():
                 stream_log_request = StreamLogParameters(**body)
         except json.JSONDecodeError:
-            # Body as text
-            validation_exception = RequestValidationError(errors=["Invalid JSON body"])
-            err_event = {
-                "event": "error",
-                "data": json.dumps(
-                    {"status_code": 422, "message": "Invalid JSON body"}
-                ),
-            }
-        except RequestValidationError as e:
-            validation_exception = e
-            err_event = {
-                "event": "error",
-                "data": json.dumps({"status_code": 422, "message": repr(e.errors())}),
-            }
+            raise RequestValidationError(errors=["Invalid JSON body"])
+        except RequestValidationError:
+            raise
+
+        feedback_key: Optional[str]
+
+        if self._token_feedback_enabled:
+            # Create task to create a presigned feedback token
+            feedback_key: str = self._token_feedback_config["key_configs"][0]["key"]
+            feedback_coro = run_in_executor(
+                None,
+                self._langsmith_client.create_presigned_feedback_token,
+                run_id,
+                feedback_key,
+            )
+            task: Optional[asyncio.Task] = asyncio.create_task(feedback_coro)
+        else:
+            feedback_key = None
+            task = None
 
         async def _stream_log() -> AsyncIterator[dict]:
             """Stream the output of the runnable."""
-            if validation_exception:
-                yield err_event
-                if isinstance(validation_exception, RequestValidationError):
-                    return
-                else:
-                    raise AssertionError(
-                        "Internal server error"
-                    ) from validation_exception
-
+            has_sent_metadata = False
             try:
                 async for chunk in self._runnable.astream_log(
                     input_,
@@ -1105,6 +1243,18 @@ class APIHandler:
                             "data": self._serializer.dumps(data).decode("utf-8"),
                             "event": "data",
                         }
+
+                    # Send a metadata event as soon as possible
+                    if not has_sent_metadata and self._token_feedback_enabled:
+                        if task is None:
+                            raise AssertionError("Feedback token task was not created.")
+                        if not task.done():
+                            continue
+                        feedback_token = task.result()
+                        yield _create_metadata_event(
+                            run_id, feedback_key, feedback_token
+                        )
+                        has_sent_metadata = True
                 yield {"event": "end"}
             except BaseException:
                 yield {
@@ -1128,8 +1278,7 @@ class APIHandler:
         server_config: Optional[RunnableConfig] = None,
     ) -> EventSourceResponse:
         """Stream events from the runnable."""
-        err_event = {}
-        validation_exception: Optional[BaseException] = None
+        run_id = None
         try:
             config, input_ = await self._get_config_and_input(
                 request,
@@ -1137,44 +1286,36 @@ class APIHandler:
                 endpoint="stream_events",
                 server_config=server_config,
             )
-        except BaseException as e:
-            validation_exception = e
-            if isinstance(e, RequestValidationError):
-                err_event = {
-                    "event": "error",
-                    "data": json.dumps(
-                        {"status_code": 422, "message": repr(e.errors())}
-                    ),
-                }
-            else:
-                err_event = {
-                    "event": "error",
-                    # Do not expose the error message to the client since
-                    # the message may contain sensitive information.
-                    "data": json.dumps(
-                        {"status_code": 500, "message": "Internal Server Error"}
-                    ),
-                }
+            run_id = config["run_id"]
+        except BaseException:
+            # Exceptions will be properly translated by default FastAPI middleware
+            # to either 422 (on input validation) or 500 internal server errors.
+            raise
 
         try:
             body = await request.json()
             with _with_validation_error_translation():
                 stream_events_request = StreamEventsParameters(**body)
         except json.JSONDecodeError:
-            # Body as text
-            validation_exception = RequestValidationError(errors=["Invalid JSON body"])
-            err_event = {
-                "event": "error",
-                "data": json.dumps(
-                    {"status_code": 422, "message": "Invalid JSON body"}
-                ),
-            }
-        except RequestValidationError as e:
-            validation_exception = e
-            err_event = {
-                "event": "error",
-                "data": json.dumps({"status_code": 422, "message": repr(e.errors())}),
-            }
+            raise RequestValidationError(errors=["Invalid JSON body"])
+        except RequestValidationError:
+            raise
+
+        feedback_key: Optional[str]
+
+        if self._token_feedback_enabled:
+            # Create task to create a presigned feedback token
+            feedback_key: str = self._token_feedback_config["key_configs"][0]["key"]
+            feedback_coro = run_in_executor(
+                None,
+                self._langsmith_client.create_presigned_feedback_token,
+                run_id,
+                feedback_key,
+            )
+            task: Optional[asyncio.Task] = asyncio.create_task(feedback_coro)
+        else:
+            feedback_key = None
+            task = None
 
         async def _stream_events() -> AsyncIterator[dict]:
             """Stream the output of the runnable."""
@@ -1183,14 +1324,7 @@ class APIHandler:
                     "Please upgrade langchain-core>=0.1.14 to use astream_events"
                 )
 
-            if validation_exception:
-                yield err_event
-                if isinstance(validation_exception, RequestValidationError):
-                    return
-                else:
-                    raise AssertionError(
-                        "Internal server error"
-                    ) from validation_exception
+            has_sent_metadata = False
 
             try:
                 async for event in self._runnable.astream_events(
@@ -1218,6 +1352,18 @@ class APIHandler:
                             "data": self._serializer.dumps(event).decode("utf-8"),
                             "event": "data",
                         }
+
+                    # Send a metadata event as soon as possible
+                    if not has_sent_metadata and self._token_feedback_enabled:
+                        if task is None:
+                            raise AssertionError("Feedback token task was not created.")
+                        if not task.done():
+                            continue
+                        feedback_token = task.result()
+                        yield _create_metadata_event(
+                            run_id, feedback_key, feedback_token
+                        )
+                        has_sent_metadata = True
                 yield {"event": "end"}
             except BaseException:
                 yield {
@@ -1360,59 +1506,13 @@ class APIHandler:
         return await serve_playground(
             self._runnable.with_config(config),
             self._runnable.with_config(config).input_schema,
+            self._runnable.with_config(config).output_schema,
             self._config_keys,
             playground_url,
             file_path,
             feedback_enabled,
             public_trace_link_enabled,
-        )
-
-    async def chat_playground(
-        self,
-        file_path: str,
-        request: Request,
-        *,
-        config_hash: str = "",
-        server_config: Optional[RunnableConfig] = None,
-    ) -> Any:
-        """Return the playground of the runnable."""
-        with _with_validation_error_translation():
-            user_provided_config = await _unpack_request_config(
-                config_hash,
-                config_keys=self._config_keys,
-                model=self._ConfigPayload,
-                request=request,
-                # Do not use per request config modifier for output schema
-                # since it's unclear why it would make sense to modify
-                # this using a per request config modifier.
-                # If this is needed, for some reason please file an issue explaining
-                # the user case.
-                per_req_config_modifier=None,
-                server_config=server_config,
-            )
-
-            config = _update_config_with_defaults(
-                self._run_name, user_provided_config, request
-            )
-
-        chat_playground_url = (
-            request.scope.get("root_path", "").rstrip("/")
-            + self._base_url
-            + "/chat_playground"
-        )
-        feedback_enabled = tracing_is_enabled() and self._enable_feedback_endpoint
-        public_trace_link_enabled = (
-            tracing_is_enabled() and self._enable_public_trace_link_endpoint
-        )
-
-        return await serve_chat_playground(
-            self._runnable.with_config(config),
-            self._runnable.with_config(config).input_schema,
-            self._config_keys,
-            chat_playground_url,
-            file_path,
-            feedback_enabled,
-            public_trace_link_enabled,
+            playground_type=self.playground_type,
         )
 
     async def create_feedback(
@@ -1461,6 +1561,38 @@ class APIHandler:
             comment=feedback_from_langsmith.comment,
         )
 
+    async def create_feedback_from_token(
+        self, create_request: FeedbackCreateRequestTokenBased
+    ) -> None:
+        """Send feedback on an individual run to langsmith."""
+
+        if not tracing_is_enabled() or not self._token_feedback_enabled:
+            raise HTTPException(
+                400,
+                (
+                    "The feedback endpoint is only accessible when LangSmith is "
+                    "enabled on your LangServe server.\n "
+                    "Please set the proper environment variables to enable LangSmith."
+                    "In addition, please ensure that the token_feedback_config "
+                    "has been properly specified when using add_routes or APIHandler."
+                ),
+            )
+
+        metadata = create_request.metadata or {}
+        metadata.update(
+            {
+                "from_langserve": True,
+            }
+        )
+
+        self._langsmith_client.create_feedback_from_token(
+            create_request.token_or_url,
+            score=create_request.score,
+            value=create_request.value,
+            comment=create_request.comment,
+            metadata=metadata,
+        )
+
     async def _check_feedback_enabled(self) -> None:
         """Check if feedback is enabled for the runnable.
 

langserve/callbacks.py

@@ -4,13 +4,16 @@ import uuid
 from typing import Any, Dict, List, Optional, Sequence
 from uuid import UUID
 
-from langchain.callbacks.base import AsyncCallbackHandler
-from langchain.callbacks.manager import (
+from langchain_core.agents import AgentAction, AgentFinish
+from langchain_core.callbacks import AsyncCallbackHandler
+from langchain_core.callbacks.manager import (
     BaseRunManager,
     ahandle_event,
     handle_event,
 )
-from langchain.schema import AgentAction, AgentFinish, BaseMessage, Document, LLMResult
+from langchain_core.documents import Document
+from langchain_core.messages import BaseMessage
+from langchain_core.outputs import LLMResult
 from typing_extensions import TypedDict
 
 

langserve/chat_playground.py

@@ -1,98 +0,0 @@
-import json
-import mimetypes
-import os
-from string import Template
-from typing import Sequence, Type
-
-from fastapi.responses import Response
-from langchain.schema.runnable import Runnable
-
-from langserve.pydantic_v1 import BaseModel
-
-
-class ChatPlaygroundTemplate(Template):
-    delimiter = "____"
-
-
-def _get_mimetype(path: str) -> str:
-    """Get mimetype for file.
-
-    Custom implementation of mimetypes.guess_type that
-    uses the file extension to determine the mimetype for some files.
-
-    This is necessary due to: https://bugs.python.org/issue43975
-    Resolves issue: https://github.com/langchain-ai/langserve/issues/245
-
-    Args:
-        path (str): Path to file
-
-    Returns:
-        str: Mimetype of file
-    """
-    try:
-        file_extension = path.lower().split(".")[-1]
-    except IndexError:
-        return mimetypes.guess_type(path)[0]
-
-    if file_extension == "js":
-        return "application/javascript"
-    elif file_extension == "css":
-        return "text/css"
-    elif file_extension in ["htm", "html"]:
-        return "text/html"
-
-    # If the file extension is not one of the specified ones,
-    # use the default guess method
-    mime_type = mimetypes.guess_type(path)[0]
-    return mime_type
-
-
-async def serve_chat_playground(
-    runnable: Runnable,
-    input_schema: Type[BaseModel],
-    config_keys: Sequence[str],
-    base_url: str,
-    file_path: str,
-    feedback_enabled: bool,
-    public_trace_link_enabled: bool,
-) -> Response:
-    """Serve the playground."""
-    local_file_path = os.path.abspath(
-        os.path.join(
-            os.path.dirname(__file__),
-            "./chat_playground/dist",
-            file_path or "index.html",
-        )
-    )
-
-    base_dir = os.path.abspath(
-        os.path.join(os.path.dirname(__file__), "./chat_playground/dist")
-    )
-
-    if base_dir != os.path.commonpath((base_dir, local_file_path)):
-        return Response("Not Found", status_code=404)
-    try:
-        with open(local_file_path, encoding="utf-8") as f:
-            mime_type = _get_mimetype(local_file_path)
-            if mime_type in ("text/html", "text/css", "application/javascript"):
-                response = ChatPlaygroundTemplate(f.read()).substitute(
-                    LANGSERVE_BASE_URL=base_url[1:]
-                    if base_url.startswith("/")
-                    else base_url,
-                    LANGSERVE_CONFIG_SCHEMA=json.dumps(
-                        runnable.config_schema(include=config_keys).schema()
-                    ),
-                    LANGSERVE_INPUT_SCHEMA=json.dumps(input_schema.schema()),
-                    LANGSERVE_FEEDBACK_ENABLED=json.dumps(
-                        "true" if feedback_enabled else "false"
-                    ),
-                    LANGSERVE_PUBLIC_TRACE_LINK_ENABLED=json.dumps(
-                        "true" if public_trace_link_enabled else "false"
-                    ),
-                )
-            else:
-                response = f.buffer.read()
-    except FileNotFoundError:
-        return Response("Not Found", status_code=404)
-
-    return Response(response, media_type=mime_type)

langserve/chat_playground/dist/index.html

@@ -5,18 +5,35 @@
     <link rel="icon" href="/____LANGSERVE_BASE_URL/favicon.ico" />
     <meta name="viewport" content="width=device-width, initial-scale=1.0" />
     <title>Chat Playground</title>
-    <script type="module" crossorigin src="/____LANGSERVE_BASE_URL/assets/index-34f4a840.js"></script>
-    <link rel="stylesheet" href="/____LANGSERVE_BASE_URL/assets/index-b47ed17e.css">
+    <script type="module" crossorigin src="/____LANGSERVE_BASE_URL/assets/index-86d4d9c0.js"></script>
+    <link rel="stylesheet" href="/____LANGSERVE_BASE_URL/assets/index-434ff580.css">
   </head>
   <body>
     <div id="root"></div>
     <script>
       try {
         window.CONFIG_SCHEMA = ____LANGSERVE_CONFIG_SCHEMA;
+      } catch (e) {
+        // pass
+      }
+      try {
         window.INPUT_SCHEMA = ____LANGSERVE_INPUT_SCHEMA;
+      } catch (e) {
+        // pass
+      }
+      try {
+        window.OUTPUT_SCHEMA = ____LANGSERVE_OUTPUT_SCHEMA;
+      } catch (e) {
+        // pass
+      }
+      try {
         window.FEEDBACK_ENABLED = ____LANGSERVE_FEEDBACK_ENABLED;
+      } catch (e) {
+        // pass
+      }
+      try {
         window.PUBLIC_TRACE_LINK_ENABLED = ____LANGSERVE_PUBLIC_TRACE_LINK_ENABLED;
-      } catch (error) {
+      } catch (e) {
         // pass
       }
     </script>

langserve/chat_playground/index.html

@@ -11,10 +11,27 @@
     <script>
       try {
         window.CONFIG_SCHEMA = ____LANGSERVE_CONFIG_SCHEMA;
+      } catch (e) {
+        // pass
+      }
+      try {
         window.INPUT_SCHEMA = ____LANGSERVE_INPUT_SCHEMA;
+      } catch (e) {
+        // pass
+      }
+      try {
+        window.OUTPUT_SCHEMA = ____LANGSERVE_OUTPUT_SCHEMA;
+      } catch (e) {
+        // pass
+      }
+      try {
         window.FEEDBACK_ENABLED = ____LANGSERVE_FEEDBACK_ENABLED;
+      } catch (e) {
+        // pass
+      }
+      try {
         window.PUBLIC_TRACE_LINK_ENABLED = ____LANGSERVE_PUBLIC_TRACE_LINK_ENABLED;
-      } catch (error) {
+      } catch (e) {
         // pass
       }
     </script>

langserve/chat_playground/src/App.tsx

@@ -2,32 +2,66 @@ import "./App.css";
 
 import { ChatWindow } from "./components/ChatWindow";
 import { AppCallbackContext, useAppStreamCallbacks } from "./useStreamCallback";
-import { useInputSchema } from "./useSchemas";
+import { useInputSchema, useOutputSchema } from "./useSchemas";
 import { useStreamLog } from "./useStreamLog";
-import { resolveApiUrl } from "./utils/url";
 
 export function App() {
   const { context, callbacks } = useAppStreamCallbacks();
   const { startStream, stopStream } = useStreamLog(callbacks);
   const inputSchema = useInputSchema({});
+  const outputSchema = useOutputSchema({});
   const inputProps = inputSchema?.data?.schema?.properties;
-  const isLoading = inputProps === undefined;
+  const outputDataSchema = outputSchema?.data?.schema;
+  const isLoading = inputProps === undefined || outputDataSchema === undefined;
   const inputKeys = Object.keys(inputProps ?? {});
-  const isSupported = isLoading || (inputKeys.length === 1 && inputProps[inputKeys[0]].type === "array");
+  const inputSchemaSupported = (
+    inputKeys.length === 1 &&
+    inputProps?.[inputKeys[0]].type === "array"
+  ) || (
+    inputKeys.length === 2 && (
+      (
+        inputProps?.[inputKeys[0]].type === "array" ||
+        inputProps?.[inputKeys[1]].type === "string"
+      ) || (
+        inputProps?.[inputKeys[0]].type === "string" ||
+        inputProps?.[inputKeys[1]].type === "array" 
+      )
+    )
+  );
+  const outputSchemaSupported = (
+    outputDataSchema?.anyOf?.find((option) => option.properties?.type?.enum?.includes("ai")) ||
+    outputDataSchema?.type === "string"
+  );
+  const isSupported = isLoading || (inputSchemaSupported && outputSchemaSupported);
   return (
     <div className="flex items-center flex-col text-ls-black bg-background">
       <AppCallbackContext.Provider value={context}>
-        {isSupported 
-          ? <ChatWindow 
+        {isSupported
+          ? <ChatWindow
               startStream={startStream}
               stopStream={stopStream}
-              inputKey={inputKeys[0]}
-            ></ChatWindow> 
-          : <div className="h-[100vh] w-[100vw] flex justify-center items-center text-xl">
-              <span className="text-center">
-                The chat playground is only supported for chains that take a single array of messages as input.
-                <br/>
-                You can test this chain in the standard <a href={resolveApiUrl("/playground").toString()}>LangServe playground</a>.
+              messagesInputKey={inputProps?.[inputKeys[0]].type === "array" ? inputKeys[0] : inputKeys[1]}
+              inputKey={inputProps?.[inputKeys[0]].type === "string" ? inputKeys[0] : inputKeys[1]}
+            ></ChatWindow>
+          : <div className="h-[100vh] w-[100vw] flex justify-center items-center text-xl p-16">
+              <span>
+                The chat playground is only supported for chains that take one of the following as input:
+                <ul className="mt-8 list-disc ml-6">
+                  <li>
+                    a dict with a single key containing a list of messages
+                  </li>
+                  <li>
+                    a dict with two keys: one a string input, one an list of messages
+                  </li>
+                </ul>
+                <br />
+                and which return either an <code>AIMessage</code> or a string.
+                <br />
+                <br />
+                You can test this chain in the default LangServe playground instead.
+                <br />
+                <br />
+                To use the default playground, set <code>playground_type="default"</code> when adding the route in your backend.
               </span>
             </div>}
       </AppCallbackContext.Provider>

langserve/chat_playground/src/components/ChatWindow.tsx

@@ -33,9 +33,10 @@ export function isAIMessage(x: unknown): x is AIMessage {
 export function ChatWindow(props: {
   startStream: (input: unknown, config: unknown) => Promise<void>;
   stopStream: (() => void) | undefined;
-  inputKey: string;
+  messagesInputKey: string;
+  inputKey?: string;
 }) {
-  const { startStream, inputKey } = props;
+  const { startStream, messagesInputKey, inputKey } = props;
 
   const [currentInputValue, setCurrentInputValue] = useState("");
   const [isLoading, setIsLoading] = useState(false);
@@ -58,7 +59,14 @@ export function ChatWindow(props: {
     setMessages(newMessages);
     setCurrentInputValue("");
     // TODO: Add config schema support
-    startStream({ [inputKey]: newMessages }, {});
+    if (inputKey === undefined) {
+      startStream({ [messagesInputKey]: newMessages }, {});
+    } else {
+      startStream({
+        [messagesInputKey]: newMessages.slice(0, -1),
+        [inputKey]: newMessages[newMessages.length - 1].content
+      }, {});
+    }
   };
 
   const regenerateMessages = () => {
@@ -67,7 +75,14 @@ export function ChatWindow(props: {
     }
     setIsLoading(true);
     // TODO: Add config schema support
-    startStream({ [inputKey]: messages }, {});
+    if (inputKey === undefined) {
+      startStream({ [messagesInputKey]: messages }, {});
+    } else {
+      startStream({
+        [messagesInputKey]: messages.slice(0, -1),
+        [inputKey]: messages[messages.length - 1].content
+      }, {});
+    }
   };
   
   useStreamCallback("onStart", () => {

langserve/chat_playground/src/components/ShareDialog.tsx

@@ -51,8 +51,8 @@ export function ShareDialog(props: { config: unknown; children: ReactNode }) {
   // get base URL
   const targetUrl = `${state.basePath}/c/${hash}`;
 
-  // .../c/[hash]/chat_playground
-  const playgroundUrl = `${targetUrl}/chat_playground`;
+  // .../c/[hash]/playground
+  const playgroundUrl = `${targetUrl}/playground`;
 
   // cURL, JS: .../c/[hash]/invoke
   // Python: .../c/[hash]

langserve/chat_playground/src/useSchemas.tsx

@@ -1,7 +1,7 @@
-import { resolveApiUrl } from "./utils/url";
-import { simplifySchema } from "./utils/simplifySchema";
 import { JsonSchema } from "@jsonforms/core";
 import { compressToEncodedURIComponent } from "lz-string";
+import { resolveApiUrl } from "./utils/url";
+import { simplifySchema } from "./utils/simplifySchema";
 
 import useSWR from "swr";
 import defaults from "./utils/defaults";
@@ -13,6 +13,8 @@ declare global {
     // eslint-disable-next-line @typescript-eslint/no-explicit-any
     INPUT_SCHEMA?: any;
     // eslint-disable-next-line @typescript-eslint/no-explicit-any
+    OUTPUT_SCHEMA?: any;
+    // eslint-disable-next-line @typescript-eslint/no-explicit-any
     FEEDBACK_ENABLED?: any;
     // eslint-disable-next-line @typescript-eslint/no-explicit-any
     PUBLIC_TRACE_LINK_ENABLED?: any;
@@ -96,3 +98,34 @@ export function useInputSchema(configData?: unknown) {
     { keepPreviousData: true }
   );
 }
+
+export function useOutputSchema(configData?: unknown) {
+  return useSWR(
+    ["/output_schema", configData],
+    async ([, configData]) => {
+      // TODO: this won't work if we're already seeing a prefixed URL
+      const prefix = configData
+        ? `/c/${compressToEncodedURIComponent(JSON.stringify(configData))}`
+        : "";
+
+      let schema: JsonSchema | null = null;
+
+      if (!prefix && !import.meta.env.DEV && window.OUTPUT_SCHEMA) {
+        schema = await simplifySchema(window.OUTPUT_SCHEMA);
+      } else {
+        const response = await fetch(resolveApiUrl(`${prefix}/output_schema`));
+        if (!response.ok) throw new Error(await response.text());
+
+        const json = await response.json();
+        schema = await simplifySchema(json);
+      }
+
+      if (schema == null) return null;
+      return {
+        schema,
+        defaults: defaults(schema),
+      };
+    },
+    { keepPreviousData: true }
+  );
+}

langserve/chat_playground/src/utils/url.ts

@@ -7,8 +7,8 @@ export function getStateFromUrl(path: string) {
     basePath = basePath.slice(0, -1);
   }
 
-  if (basePath.endsWith("/chat_playground")) {
-    basePath = basePath.slice(0, -"/chat_playground".length);
+  if (basePath.endsWith("/playground")) {
+    basePath = basePath.slice(0, -"/playground".length);
   }
 
   // check if we can omit the last segment

langserve/chat_playground/vite.config.ts

@@ -8,7 +8,7 @@ export default defineConfig({
   plugins: [svgr(), react()],
   server: {
     proxy: {
-      "^/____LANGSERVE_BASE_URL.*/(config_schema|input_schema|stream_log|feedback|public_trace_link)(/[a-zA-Z0-9-]*)?$": {
+      "^/____LANGSERVE_BASE_URL.*/(config_schema|input_schema|output_schema|stream_log|feedback|public_trace_link)(/[a-zA-Z0-9-]*)?$": {
         target: "http://127.0.0.1:8000",
         changeOrigin: true,
         rewrite: (path) => path.replace("/____LANGSERVE_BASE_URL", ""),

langserve/client.py

@@ -22,21 +22,21 @@ from urllib.parse import urljoin
 
 import httpx
 from httpx._types import AuthTypes, CertTypes, CookieTypes, HeaderTypes, VerifyTypes
-from langchain.callbacks.manager import (
+from langchain_core.callbacks import (
     AsyncCallbackManagerForChainRun,
     CallbackManagerForChainRun,
 )
-from langchain.callbacks.tracers.log_stream import RunLogPatch
-from langchain.load.dump import dumpd
-from langchain.schema.runnable import Runnable
-from langchain.schema.runnable.config import (
+from langchain_core.load.dump import dumpd
+from langchain_core.runnables import Runnable
+from langchain_core.runnables.config import (
     RunnableConfig,
     ensure_config,
     get_async_callback_manager_for_config,
     get_callback_manager_for_config,
 )
-from langchain.schema.runnable.utils import AddableDict, Input, Output
 from langchain_core.runnables.schema import StreamEvent
+from langchain_core.runnables.utils import AddableDict, Input, Output
+from langchain_core.tracers.log_stream import RunLogPatch
 from typing_extensions import Literal
 
 from langserve.callbacks import CallbackEventDict, ahandle_callbacks, handle_callbacks
@@ -45,6 +45,7 @@ from langserve.serialization import (
     WellKnownLCSerializer,
     load_events,
 )
+from langserve.server_sent_events import aconnect_sse, connect_sse
 
 logger = logging.getLogger(__name__)
 
@@ -523,25 +524,17 @@ class RemoteRunnable(Runnable[Input, Output]):
         }
         endpoint = urljoin(self.url, "stream")
 
-        try:
-            from httpx_sse import connect_sse
-        except ImportError:
-            raise ImportError(
-                "Missing `httpx_sse` dependency to use the stream method. "
-                "Install via `pip install httpx_sse`'"
-            )
-
         try:
             with connect_sse(
                 self.sync_client, "POST", endpoint, json=data
             ) as event_source:
                 for sse in event_source.iter_sse():
-                    if sse.event == "data":
-                        chunk = self._lc_serializer.loads(sse.data)
+                    if sse["event"] == "data":
+                        chunk = self._lc_serializer.loads(sse["data"])
                         if isinstance(chunk, dict):
                             # Any dict returned from streaming end point
                             # is assumed to follow additive semantics
-                            # and will be converted to an AddableDict
+                            # and will be coverted to an AddableDict
                             # automatically
                             chunk = AddableDict(chunk)
                         yield chunk
@@ -563,21 +556,21 @@ class RemoteRunnable(Runnable[Input, Output]):
                                 except TypeError:
                                     final_output = None
                                     final_output_supported = False
-                    elif sse.event == "error":
+                    elif sse["event"] == "error":
                         # This can only be a server side error
                         _raise_exception_from_data(
-                            sse.data, httpx.Request(method="POST", url=endpoint)
+                            sse["data"], httpx.Request(method="POST", url=endpoint)
                         )
-                    elif sse.event == "metadata":
+                    elif sse["event"] == "metadata":
                         # Nothing to do for metadata for the regular remote client.
                         continue
-                    elif sse.event == "end":
+                    elif sse["event"] == "end":
                         break
                     else:
                         _log_error_message_once(
-                            f"Encountered an unsupported event type: `{sse.event}`. "
+                            f"Encountered an unsupported event type: `{sse['event']}`. "
                             f"Try upgrading the remote client to the latest version."
-                            f"Ignoring events of type `{sse.event}`."
+                            f"Ignoring events of type `{sse['event']}`."
                         )
         except BaseException as e:
             run_manager.on_chain_error(e)
@@ -609,18 +602,13 @@ class RemoteRunnable(Runnable[Input, Output]):
         }
         endpoint = urljoin(self.url, "stream")
 
-        try:
-            from httpx_sse import aconnect_sse
-        except ImportError:
-            raise ImportError("You must install `httpx_sse` to use the stream method.")
-
         try:
             async with aconnect_sse(
                 self.async_client, "POST", endpoint, json=data
             ) as event_source:
                 async for sse in event_source.aiter_sse():
-                    if sse.event == "data":
-                        chunk = self._lc_serializer.loads(sse.data)
+                    if sse["event"] == "data":
+                        chunk = self._lc_serializer.loads(sse["data"])
                         if isinstance(chunk, dict):
                             # Any dict returned from streaming end point
                             # is assumed to follow additive semantics
@@ -647,21 +635,21 @@ class RemoteRunnable(Runnable[Input, Output]):
                                     final_output = None
                                     final_output_supported = False
 
-                    elif sse.event == "error":
+                    elif sse["event"] == "error":
                         # This can only be a server side error
                         _raise_exception_from_data(
-                            sse.data, httpx.Request(method="POST", url=endpoint)
+                            sse["data"], httpx.Request(method="POST", url=endpoint)
                         )
-                    elif sse.event == "metadata":
+                    elif sse["event"] == "metadata":
                         # Nothing to do for metadata for the regular remote client.
                         continue
-                    elif sse.event == "end":
+                    elif sse["event"] == "end":
                         break
                     else:
                         _log_error_message_once(
-                            f"Encountered an unsupported event type: `{sse.event}`. "
+                            f"Encountered an unsupported event type: `{sse['event']}`. "
                             f"Try upgrading the remote client to the latest version."
-                            f"Ignoring events of type `{sse.event}`."
+                            f"Ignoring events of type `{sse['event']}`."
                         )
         except BaseException as e:
             await run_manager.on_chain_error(e)
@@ -718,18 +706,13 @@ class RemoteRunnable(Runnable[Input, Output]):
         }
         endpoint = urljoin(self.url, "stream_log")
 
-        try:
-            from httpx_sse import aconnect_sse
-        except ImportError:
-            raise ImportError("You must install `httpx_sse` to use the stream method.")
-
         try:
             async with aconnect_sse(
                 self.async_client, "POST", endpoint, json=data
             ) as event_source:
                 async for sse in event_source.aiter_sse():
-                    if sse.event == "data":
-                        data = self._lc_serializer.loads(sse.data)
+                    if sse["event"] == "data":
+                        data = self._lc_serializer.loads(sse["data"])
                         # Create a copy of the data to yield since underlying
                         # code is using jsonpatch which does some stuff in-place
                         # that can cause unexpected consequences.
@@ -741,18 +724,18 @@ class RemoteRunnable(Runnable[Input, Output]):
                             final_output += chunk
                         else:
                             final_output = chunk
-                    elif sse.event == "error":
+                    elif sse["event"] == "error":
                         # This can only be a server side error
                         _raise_exception_from_data(
-                            sse.data, httpx.Request(method="POST", url=endpoint)
+                            sse["data"], httpx.Request(method="POST", url=endpoint)
                         )
-                    elif sse.event == "end":
+                    elif sse["event"] == "end":
                         break
                     else:
                         _log_error_message_once(
-                            f"Encountered an unsupported event type: `{sse.event}`. "
+                            f"Encountered an unsupported event type: `{sse['event']}`. "
                             f"Try upgrading the remote client to the latest version."
-                            f"Ignoring events of type `{sse.event}`."
+                            f"Ignoring events of type `{sse['event']}`."
                         )
         except BaseException as e:
             await run_manager.on_chain_error(e)
@@ -822,36 +805,34 @@ class RemoteRunnable(Runnable[Input, Output]):
             "exclude_tags": exclude_tags,
         }
         endpoint = urljoin(self.url, "stream_events")
-
-        try:
-            from httpx_sse import aconnect_sse
-        except ImportError:
-            raise ImportError("You must install `httpx_sse` to use the stream method.")
+        headers = kwargs.pop("headers", {})
+        headers["Accept"] = "text/event-stream"
+        headers["Cache-Control"] = "no-store"
 
         try:
             async with aconnect_sse(
                 self.async_client, "POST", endpoint, json=data
             ) as event_source:
                 async for sse in event_source.aiter_sse():
-                    if sse.event == "data":
-                        event = self._lc_serializer.loads(sse.data)
+                    if sse["event"] == "data":
+                        event = self._lc_serializer.loads(sse["data"])
                         # Create a copy of the data to yield since underlying
                         # code is using jsonpatch which does some stuff in-place
                         # that can cause unexpected consequences.
                         yield event
                         events.append(event)
-                    elif sse.event == "error":
+                    elif sse["event"] == "error":
                         # This can only be a server side error
                         _raise_exception_from_data(
-                            sse.data, httpx.Request(method="POST", url=endpoint)
+                            sse["data"], httpx.Request(method="POST", url=endpoint)
                         )
-                    elif sse.event == "end":
+                    elif sse["event"] == "end":
                         break
                     else:
                         _log_error_message_once(
-                            f"Encountered an unsupported event type: `{sse.event}`. "
+                            f"Encountered an unsupported event type: `{sse['event']}`. "
                             f"Try upgrading the remote client to the latest version."
-                            f"Ignoring events of type `{sse.event}`."
+                            f"Ignoring events of type `{sse['event']}`."
                         )
         except BaseException as e:
             await run_manager.on_chain_error(e)

langserve/playground.py

@@ -2,10 +2,10 @@ import json
 import mimetypes
 import os
 from string import Template
-from typing import Sequence, Type
+from typing import Literal, Sequence, Type
 
 from fastapi.responses import Response
-from langchain.schema.runnable import Runnable
+from langchain_core.runnables import Runnable
 
 from langserve.pydantic_v1 import BaseModel
 
@@ -50,28 +50,37 @@ def _get_mimetype(path: str) -> str:
 async def serve_playground(
     runnable: Runnable,
     input_schema: Type[BaseModel],
+    output_schema: Type[BaseModel],
     config_keys: Sequence[str],
     base_url: str,
     file_path: str,
     feedback_enabled: bool,
     public_trace_link_enabled: bool,
+    playground_type: Literal["default", "chat"],
 ) -> Response:
     """Serve the playground."""
+    if playground_type == "default":
+        path_to_dist = "./playground/dist"
+    elif playground_type == "chat":
+        path_to_dist = "./chat_playground/dist"
+    else:
+        raise ValueError(
+            f"Invalid playground type: {playground_type}. "
+            f"Use one of 'default' or 'chat'."
+        )
+
     local_file_path = os.path.abspath(
         os.path.join(
             os.path.dirname(__file__),
-            "./playground/dist",
+            path_to_dist,
             file_path or "index.html",
         )
     )
 
-    base_dir = os.path.abspath(
-        os.path.join(os.path.dirname(__file__), "./playground/dist")
-    )
+    base_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), path_to_dist))
 
     if base_dir != os.path.commonpath((base_dir, local_file_path)):
         return Response("Not Found", status_code=404)
-
     try:
         with open(local_file_path, encoding="utf-8") as f:
             mime_type = _get_mimetype(local_file_path)
@@ -84,6 +93,7 @@ async def serve_playground(
                         runnable.config_schema(include=config_keys).schema()
                     ),
                     LANGSERVE_INPUT_SCHEMA=json.dumps(input_schema.schema()),
+                    LANGSERVE_OUTPUT_SCHEMA=json.dumps(output_schema.schema()),
                     LANGSERVE_FEEDBACK_ENABLED=json.dumps(
                         "true" if feedback_enabled else "false"
                     ),

langserve/playground/dist/index.html

@@ -14,6 +14,7 @@
       try {
         window.CONFIG_SCHEMA = ____LANGSERVE_CONFIG_SCHEMA;
         window.INPUT_SCHEMA = ____LANGSERVE_INPUT_SCHEMA;
+        window.OUTPUT_SCHEMA = ____LANGSERVE_OUTPUT_SCHEMA;
         window.FEEDBACK_ENABLED = ____LANGSERVE_FEEDBACK_ENABLED;
         window.PUBLIC_TRACE_LINK_ENABLED = ____LANGSERVE_PUBLIC_TRACE_LINK_ENABLED;
       } catch (error) {

langserve/playground/index.html

@@ -12,6 +12,7 @@
       try {
         window.CONFIG_SCHEMA = ____LANGSERVE_CONFIG_SCHEMA;
         window.INPUT_SCHEMA = ____LANGSERVE_INPUT_SCHEMA;
+        window.OUTPUT_SCHEMA = ____LANGSERVE_OUTPUT_SCHEMA;
         window.FEEDBACK_ENABLED = ____LANGSERVE_FEEDBACK_ENABLED;
         window.PUBLIC_TRACE_LINK_ENABLED = ____LANGSERVE_PUBLIC_TRACE_LINK_ENABLED;
       } catch (error) {

langserve/playground/src/useSchemas.tsx

@@ -13,7 +13,11 @@ declare global {
     // eslint-disable-next-line @typescript-eslint/no-explicit-any
     INPUT_SCHEMA?: any;
     // eslint-disable-next-line @typescript-eslint/no-explicit-any
+    OUTPUT_SCHEMA?: any;
+    // eslint-disable-next-line @typescript-eslint/no-explicit-any
     FEEDBACK_ENABLED?: any;
+    // eslint-disable-next-line @typescript-eslint/no-explicit-any
+    PUBLIC_TRACE_LINK_ENABLED?: any;
   }
 }
 

langserve/schema.py

@@ -5,6 +5,7 @@ from uuid import UUID
 from pydantic import BaseModel  # Floats between v1 and v2
 
 from langserve.pydantic_v1 import BaseModel as BaseModelV1
+from langserve.pydantic_v1 import Field
 
 
 class CustomUserType(BaseModelV1):
@@ -41,14 +42,35 @@ class SharedResponseMetadata(BaseModelV1):
     pass
 
 
-class SingletonResponseMetadata(SharedResponseMetadata):
-    """
-    Represents response metadata used for just single input/output LangServe
+class FeedbackToken(BaseModelV1):
+    """Represents the feedback tokens for a given request."""
+
+    key: str  # The key of the feedback token
+    token_url: Optional[str] = None
+    expires_at: Optional[datetime] = None
+
+
+class InvokeResponseMetadata(SharedResponseMetadata):
+    """Represents response metadata used for just single input/output LangServe
     responses.
     """
 
     # Represents the parent run id for a given request
     run_id: UUID
+    feedback_tokens: List[FeedbackToken] = Field(
+        ...,
+        description=(
+            "Feedback tokens from the given run."
+            "These tokens allow a user to provide feedback on the run."
+            "Only available if server was configured to provide feedback tokens."
+        ),
+    )
+
+
+# Alias for backwards compatibility
+# Keep here in case clients are somehow using this for type checking
+# TODO(Deprecate): This should be deprecated in 2025.
+SingletonResponseMetadata = InvokeResponseMetadata
 
 
 class BatchResponseMetadata(SharedResponseMetadata):
@@ -57,17 +79,25 @@ class BatchResponseMetadata(SharedResponseMetadata):
     responses.
     """
 
+    # This namespace can include any additional metadata that is shared
+    # across all responses in the batch (e.g., if a batch run
+    # ID was a thing, it would go here)
+
+    # metadata for each individual response in the batch
+    # Parallel list of InvokeResponseMetadata objects matching
+    # the individual requests in the batch
+    responses: List[InvokeResponseMetadata]
+
+    # A list of UUIDs
     # Represents each parent run id for a given request, in
     # the same order in which they were received
-    run_ids: List[UUID]
+    run_ids: List[UUID]  # For backwards compatibility, clients should not use this
 
 
 class BaseFeedback(BaseModel):
-    """
-    Shared information between create requests of feedback and feedback objects
-    """
+    """Shared information between create requests of feedback and feedback objects"""
 
-    run_id: UUID
+    run_id: Optional[UUID]
     """The associated run ID this feedback is logged for."""
 
     key: str
@@ -83,18 +113,34 @@ class BaseFeedback(BaseModel):
     """Comment or explanation for the feedback."""
 
 
-class FeedbackCreateRequest(BaseFeedback):
-    """
-    Represents a request that creates feedback for an individual run
-    """
+class FeedbackCreateRequestTokenBased(BaseModel):
+    """Shared information between create requests of feedback and feedback objects."""
 
-    pass
+    token_or_url: Union[UUID, str]
+    """The associated run ID this feedback is logged for."""
+
+    score: Optional[Union[float, int, bool]] = None
+    """Value or score to assign the run."""
+
+    value: Optional[Union[float, int, bool, str, Dict]] = None
+    """The display value for the feedback if not a metric."""
+
+    comment: Optional[str] = None
+    """Comment or explanation for the feedback."""
+
+    correction: Optional[Dict] = None
+    """Correction for the run."""
+
+    metadata: Optional[Dict] = None
+    """Metadata for the feedback."""
+
+
+class FeedbackCreateRequest(BaseFeedback):
+    """Represents a request that creates feedback for an individual run"""
 
 
 class Feedback(BaseFeedback):
-    """
-    Represents feedback given on an individual run
-    """
+    """Represents feedback given on an individual run"""
 
     id: UUID
     """The unique ID of the feedback that was created."""
@@ -110,9 +156,7 @@ class Feedback(BaseFeedback):
 
 
 class PublicTraceLinkCreateRequest(BaseModel):
-    """
-    Represents a request that creates a public trace for an individual run
-    """
+    """Represents a request that creates a public trace for an individual run."""
 
     run_id: UUID
     """The unique ID of the run to share."""

langserve/serialization.py

@@ -16,11 +16,9 @@ from functools import lru_cache
 from typing import Any, Dict, List, Union
 
 import orjson
-from langchain.prompts.base import StringPromptValue
-from langchain.prompts.chat import ChatPromptValueConcrete
-from langchain.schema.agent import AgentAction, AgentActionMessageLog, AgentFinish
-from langchain.schema.document import Document
-from langchain.schema.messages import (
+from langchain_core.agents import AgentAction, AgentActionMessageLog, AgentFinish
+from langchain_core.documents import Document
+from langchain_core.messages import (
     AIMessage,
     AIMessageChunk,
     ChatMessage,
@@ -32,12 +30,14 @@ from langchain.schema.messages import (
     SystemMessage,
     SystemMessageChunk,
 )
-from langchain.schema.output import (
+from langchain_core.outputs import (
     ChatGeneration,
     ChatGenerationChunk,
     Generation,
     LLMResult,
 )
+from langchain_core.prompt_values import ChatPromptValueConcrete
+from langchain_core.prompts.base import StringPromptValue
 
 from langserve.pydantic_v1 import BaseModel, ValidationError
 from langserve.validation import CallbackEvent

langserve/server.py

@@ -15,10 +15,15 @@ from typing import (
     Union,
 )
 
-from langchain.schema.runnable import Runnable
+from langchain_core.runnables import Runnable
 from typing_extensions import Annotated
 
-from langserve.api_handler import APIHandler, PerRequestConfigModifier, _is_hosted
+from langserve.api_handler import (
+    APIHandler,
+    PerRequestConfigModifier,
+    TokenFeedbackConfig,
+    _is_hosted,
+)
 from langserve.pydantic_v1 import (
     _PYDANTIC_MAJOR_VERSION,
     PYDANTIC_VERSION,
@@ -39,71 +44,15 @@ except ImportError:
 # Duplicated model names break fastapi's openapi generation.
 
 _APP_SEEN = weakref.WeakSet()
+
+# Keeps track of the paths that have been associated with each app.
+# Each runnable registered with an APP will have a unique path.
+# An APP can have multiple runnables registered with it.
+# There are multiple APPs as it's common to use APIRouter in larger
+# FastAPI applications.
 _APP_TO_PATHS = weakref.WeakKeyDictionary()
 
 
-def _setup_global_app_handlers(app: Union[FastAPI, APIRouter]) -> None:
-    @app.on_event("startup")
-    async def startup_event():
-        LANGSERVE = r"""
- __          ___      .__   __.   _______      _______. _______ .______     ____    ____  _______
-|  |        /   \     |  \ |  |  /  _____|    /       ||   ____||   _  \    \   \  /   / |   ____|
-|  |       /  ^  \    |   \|  | |  |  __     |   (----`|  |__   |  |_)  |    \   \/   /  |  |__
-|  |      /  /_\  \   |  . `  | |  | |_ |     \   \    |   __|  |      /      \      /   |   __|
-|  `----./  _____  \  |  |\   | |  |__| | .----)   |   |  |____ |  |\  \----.  \    /    |  |____
-|_______/__/     \__\ |__| \__|  \______| |_______/    |_______|| _| `._____|   \__/     |_______|
-"""  # noqa: E501
-
-        def green(text: str) -> str:
-            """Return the given text in green."""
-            return "\x1b[1;32;40m" + text + "\x1b[0m"
-
-        def orange(text: str) -> str:
-            """Return the given text in orange."""
-            return "\x1b[1;31;40m" + text + "\x1b[0m"
-
-        paths = _APP_TO_PATHS[app]
-        print(LANGSERVE)
-        for path in paths:
-            print(
-                f'{green("LANGSERVE:")} Playground for chain "{path or ""}/" is '
-                f"live at:"
-            )
-            print(f'{green("LANGSERVE:")}  │')
-            print(f'{green("LANGSERVE:")}  └──> {path}/playground/')
-            print(f'{green("LANGSERVE:")}')
-        print(f'{green("LANGSERVE:")} See all available routes at {app.docs_url}/')
-
-        if _PYDANTIC_MAJOR_VERSION == 2:
-            print()
-            print(f'{orange("LANGSERVE:")} ', end="")
-            print(
-                f"⚠️ Using pydantic {PYDANTIC_VERSION}. "
-                f"OpenAPI docs for invoke, batch, stream, stream_log "
-                f"endpoints will not be generated. API endpoints and playground "
-                f"should work as expected. "
-                f"If you need to see the docs, you can downgrade to pydantic 1. "
-                "For example, `pip install pydantic==1.10.13`. "
-                f"See https://github.com/tiangolo/fastapi/issues/10360 for details."
-            )
-        print()
-
-
-def _register_path_for_app(app: Union[FastAPI, APIRouter], path: str) -> None:
-    """Register a path when its added to app. Raise if path already seen."""
-    if app in _APP_TO_PATHS:
-        seen_paths = _APP_TO_PATHS.get(app)
-        if path in seen_paths:
-            raise ValueError(
-                f"A runnable already exists at path: {path}. If adding "
-                f"multiple runnables make sure they have different paths."
-            )
-        seen_paths.add(path)
-    else:
-        _setup_global_app_handlers(app)
-        _APP_TO_PATHS[app] = {path}
-
-
 # This is the type annotation
 EndpointName = Literal[
     "invoke",
@@ -112,7 +61,6 @@ EndpointName = Literal[
     "stream_log",
     "stream_events",
     "playground",
-    "chat_playground",
     "feedback",
     "public_trace_link",
     "input_schema",
@@ -130,8 +78,8 @@ KNOWN_ENDPOINTS = {
     "stream_log",
     "stream_events",
     "playground",
-    "chat_playground",
     "feedback",
+    "token_feedback",
     "public_trace_link",
     "input_schema",
     "config_schema",
@@ -179,11 +127,11 @@ class _EndpointConfiguration:
                 is_stream_log_enabled = True
                 is_stream_events_enabled = True
                 is_playground_enabled = True
-                is_chat_playground_enabled = True
                 is_input_schema_enabled = True
                 is_output_schema_enabled = True
                 is_config_schema_enabled = True
                 is_config_hash_enabled = True
+                is_token_feedback_enabled = True
             else:
                 disabled_endpoints_ = set(name.lower() for name in disabled_endpoints)
                 if disabled_endpoints_ - KNOWN_ENDPOINTS:
@@ -197,13 +145,11 @@ class _EndpointConfiguration:
                 is_stream_log_enabled = "stream_log" not in disabled_endpoints_
                 is_stream_events_enabled = "stream_events" not in disabled_endpoints_
                 is_playground_enabled = "playground" not in disabled_endpoints_
-                is_chat_playground_enabled = (
-                    "chat_playground" not in disabled_endpoints_
-                )
                 is_input_schema_enabled = "input_schema" not in disabled_endpoints_
                 is_output_schema_enabled = "output_schema" not in disabled_endpoints_
                 is_config_schema_enabled = "config_schema" not in disabled_endpoints_
                 is_config_hash_enabled = "config_hashes" not in disabled_endpoints_
+                is_token_feedback_enabled = "token_feedback" not in disabled_endpoints_
         else:
             enabled_endpoints_ = set(name.lower() for name in enabled_endpoints)
             if enabled_endpoints_ - KNOWN_ENDPOINTS:
@@ -216,11 +162,11 @@ class _EndpointConfiguration:
             is_stream_log_enabled = "stream_log" in enabled_endpoints_
             is_stream_events_enabled = "stream_events" in enabled_endpoints_
             is_playground_enabled = "playground" in enabled_endpoints_
-            is_chat_playground_enabled = "chat_playground" in enabled_endpoints_
             is_input_schema_enabled = "input_schema" in enabled_endpoints_
             is_output_schema_enabled = "output_schema" in enabled_endpoints_
             is_config_schema_enabled = "config_schema" in enabled_endpoints_
             is_config_hash_enabled = "config_hashes" in enabled_endpoints_
+            is_token_feedback_enabled = "token_feedback" in enabled_endpoints_
 
         self.is_invoke_enabled = is_invoke_enabled
         self.is_batch_enabled = is_batch_enabled
@@ -228,13 +174,82 @@ class _EndpointConfiguration:
         self.is_stream_log_enabled = is_stream_log_enabled
         self.is_stream_events_enabled = is_stream_events_enabled
         self.is_playground_enabled = is_playground_enabled
-        self.is_chat_playground_enabled = is_chat_playground_enabled
         self.is_input_schema_enabled = is_input_schema_enabled
         self.is_output_schema_enabled = is_output_schema_enabled
         self.is_config_schema_enabled = is_config_schema_enabled
         self.is_config_hash_enabled = is_config_hash_enabled
         self.is_feedback_enabled = enable_feedback_endpoint
         self.is_public_trace_link_enabled = enable_public_trace_link_endpoint
+        self.is_token_feedback_enabled = is_token_feedback_enabled
+
+
+def _register_path_for_app(
+    app: Union[FastAPI, APIRouter],
+    path: str,
+    endpoint_configuration: _EndpointConfiguration,
+) -> None:
+    """Register a path when its added to app. Raise if path already seen."""
+    if app in _APP_TO_PATHS:
+        seen_paths = _APP_TO_PATHS.get(app)
+        if path in seen_paths:
+            raise ValueError(
+                f"A runnable already exists at path: {path}. If adding "
+                f"multiple runnables make sure they have different paths."
+            )
+        seen_paths.add(path)
+    else:
+        _setup_global_app_handlers(app, endpoint_configuration)
+        _APP_TO_PATHS[app] = {path}
+
+
+def _setup_global_app_handlers(
+    app: Union[FastAPI, APIRouter], endpoint_configuration: _EndpointConfiguration
+) -> None:
+    @app.on_event("startup")
+    async def startup_event():
+        LANGSERVE = r"""
+ __          ___      .__   __.   _______      _______. _______ .______     ____    ____  _______
+|  |        /   \     |  \ |  |  /  _____|    /       ||   ____||   _  \    \   \  /   / |   ____|
+|  |       /  ^  \    |   \|  | |  |  __     |   (----`|  |__   |  |_)  |    \   \/   /  |  |__
+|  |      /  /_\  \   |  . `  | |  | |_ |     \   \    |   __|  |      /      \      /   |   __|
+|  `----./  _____  \  |  |\   | |  |__| | .----)   |   |  |____ |  |\  \----.  \    /    |  |____
+|_______/__/     \__\ |__| \__|  \______| |_______/    |_______|| _| `._____|   \__/     |_______|
+"""  # noqa: E501
+
+        def green(text: str) -> str:
+            """Return the given text in green."""
+            return "\x1b[1;32;40m" + text + "\x1b[0m"
+
+        def orange(text: str) -> str:
+            """Return the given text in orange."""
+            return "\x1b[1;31;40m" + text + "\x1b[0m"
+
+        paths = _APP_TO_PATHS[app]
+        print(LANGSERVE)
+        for path in paths:
+            if endpoint_configuration.is_playground_enabled:
+                print(
+                    f'{green("LANGSERVE:")} Playground for chain "{path or ""}/" is '
+                    f"live at:"
+                )
+                print(f'{green("LANGSERVE:")}  │')
+                print(f'{green("LANGSERVE:")}  └──> {path}/playground/')
+                print(f'{green("LANGSERVE:")}')
+        print(f'{green("LANGSERVE:")} See all available routes at {app.docs_url}/')
+
+        if _PYDANTIC_MAJOR_VERSION == 2:
+            print()
+            print(f'{orange("LANGSERVE:")} ', end="")
+            print(
+                f"⚠️ Using pydantic {PYDANTIC_VERSION}. "
+                f"OpenAPI docs for invoke, batch, stream, stream_log "
+                f"endpoints will not be generated. API endpoints and playground "
+                f"should work as expected. "
+                f"If you need to see the docs, you can downgrade to pydantic 1. "
+                "For example, `pip install pydantic==1.10.13`. "
+                f"See https://github.com/tiangolo/fastapi/issues/10360 for details."
+            )
+        print()
 
 
 # PUBLIC API
@@ -251,11 +266,13 @@ def add_routes(
     include_callback_events: bool = False,
     per_req_config_modifier: Optional[PerRequestConfigModifier] = None,
     enable_feedback_endpoint: bool = _is_hosted(),
+    token_feedback_config: Optional[TokenFeedbackConfig] = None,
     enable_public_trace_link_endpoint: bool = False,
     disabled_endpoints: Optional[Sequence[EndpointName]] = None,
     stream_log_name_allow_list: Optional[Sequence[str]] = None,
     enabled_endpoints: Optional[Sequence[EndpointName]] = None,
     dependencies: Optional[Sequence[Depends]] = None,
+    playground_type: Literal["default", "chat"] = "default",
 ) -> None:
     """Register the routes on the given FastAPI app or APIRouter.
 
@@ -304,16 +321,26 @@ def add_routes(
             to LangSmith. Enabled by default. If this flag is disabled or LangSmith
             tracing is not enabled for the runnable, then 400 errors will be thrown
             when accessing the feedback endpoint.
+        token_feedback_config: optional configuration for token based feedback.
+            **Attention** this is distinct from `enable_feedback_endpoint`.
+            When provided, feedback tokens will be included in the response
+            metadata that can be used to provide feedback on the run.
+            In addition, an endpoint will be created for submitting feedback
+            using the feedback tokens. This is a safer option for public facing
+            APIs as they scope the feedback to a specific run id and key
+            and include an expiration time.
+            This endpoint will be created at /token_feedback
+            **BETA**: This feature is in beta and may change in the future.
         enable_public_trace_link_endpoint: Whether to enable an endpoint for
             end-users to publicly view LangSmith traces of your chain runs.
             WARNING: THIS WILL EXPOSE THE INTERNAL STATE OF YOUR RUN AND CHAIN AS
-            A PUBLICY ACCESSIBLE LINK.
+            A PUBLICLY ACCESSIBLE LINK.
             If this flag is disabled or LangSmith tracing is not enabled for
             the runnable, then 400 errors will be thrown when accessing the endpoint.
         enabled_endpoints: A list of endpoints which should be enabled. If not
             specified, all associated endpoints will be enabled. The list can contain
             the following values: *invoke*, *batch*, *stream*, *stream_log*,
-            *playground*, *chat_playground*, *input_schema*, *output_schema*,
+            *playground*, *input_schema*, *output_schema*,
             *config_schema*, *config_hashes*.
 
             *config_hashes* represents the config hash variant (when it exists)
@@ -337,7 +364,7 @@ def add_routes(
         disabled_endpoints: A list of endpoints which should be disabled. If not
             specified, all associated endpoints will be enabled. The list can contain
             the following values: *invoke*, *batch*, *stream*, *stream_log*,
-            *playground*, *chat_playground*, *input_schema*, *output_schema*,
+            *playground*, *input_schema*, *output_schema*,
             *config_schema*, *config_hashes*.
 
             *config_hashes* represents the config hash variant (when it exists)
@@ -358,7 +385,22 @@ def add_routes(
             stream as intermediate steps
         dependencies: list of dependencies to be applied to the *path operation*.
             See [FastAPI docs for Dependencies in path operation decorators](https://fastapi.tiangolo.com/tutorial/dependencies/dependencies-in-path-operation-decorators/).
+        playground_type: The type of playground to serve. The default is "default".
+            - default: supports more types of inputs / outputs. Not optimized
+              for any particular use case.
+            - chat: UX is optimized for chat-like interactions. Please review
+              the README in langserve for more details about constraints (e.g.,
+              which message types are supported etc.)
     """  # noqa: E501
+    if not isinstance(runnable, Runnable):
+        raise TypeError(
+            f"Expected a Runnable, got {type(runnable)}. "
+            f"The second argument to add_routes should be a Runnable instance."
+            f"add_route(app, runnable, ...) is the correct usage."
+            f"Please make sure that you are using a runnable which is an instance of "
+            f"langchain_core.runnables.Runnable."
+        )
+
     endpoint_configuration = _EndpointConfiguration(
         enabled_endpoints=enabled_endpoints,
         disabled_endpoints=disabled_endpoints,
@@ -384,7 +426,7 @@ def add_routes(
     if isinstance(app, FastAPI):  # type: ignore
         # Cannot do this checking logic for a router since
         # API routers are not hashable
-        _register_path_for_app(app, path)
+        _register_path_for_app(app, path, endpoint_configuration)
 
     # Determine the base URL for the playground endpoint
     prefix = app.prefix if isinstance(app, APIRouter) else ""  # type: ignore
@@ -407,9 +449,11 @@ def add_routes(
         config_keys=config_keys,
         include_callback_events=include_callback_events,
         enable_feedback_endpoint=enable_feedback_endpoint,
+        token_feedback_config=token_feedback_config,
         enable_public_trace_link_endpoint=enable_public_trace_link_endpoint,
         per_req_config_modifier=per_req_config_modifier,
         stream_log_name_allow_list=stream_log_name_allow_list,
+        playground_type=playground_type,
     )
     namespace = path or ""
 
@@ -702,19 +746,11 @@ def add_routes(
                 include_in_schema=False,
             )(playground)
 
-    if endpoint_configuration.is_chat_playground_enabled:
-        chat_playground = app.get(
-            namespace + "/chat_playground/{file_path:path}",
+    if endpoint_configuration.is_token_feedback_enabled:
+        app.post(
+            namespace + "/token_feedback",
             dependencies=dependencies,
-            include_in_schema=False,
-        )(api_handler.chat_playground)
-
-        if endpoint_configuration.is_config_hash_enabled:
-            app.get(
-                namespace + "/c/{config_hash}/chat_playground/{file_path:path}",
-                dependencies=dependencies,
-                include_in_schema=False,
-            )(chat_playground)
+        )(api_handler.create_feedback_from_token)
 
     if enable_feedback_endpoint:
         app.post(

langserve/server_sent_events.py

@@ -0,0 +1,136 @@
+"""Adapted from https://github.com/florimondmanca/httpx-sse"""
+from contextlib import asynccontextmanager, contextmanager
+from typing import Any, AsyncIterator, Iterator, List, Optional, TypedDict
+
+import httpx
+
+
+class ServerSentEvent(TypedDict):
+    event: Optional[str]
+    data: Optional[str]
+    id: Optional[str]
+    retry: Optional[int]
+
+
+class SSEDecoder:
+    def __init__(self) -> None:
+        self._event = ""
+        self._data: List[str] = []
+        self._last_event_id = ""
+        self._retry: Optional[int] = None
+
+    def decode(self, line: str) -> Optional[ServerSentEvent]:
+        # See: https://html.spec.whatwg.org/multipage/server-sent-events.html#event-stream-interpretation  # noqa: E501
+        if not line:
+            if (
+                not self._event
+                and not self._data
+                and not self._last_event_id
+                and self._retry is None
+            ):
+                return None
+
+            sse = {
+                "event": self._event,
+                "data": "\n".join(self._data),
+                "id": self._last_event_id,
+                "retry": self._retry,
+            }
+
+            # NOTE: as per the SSE spec, do not reset last_event_id.
+            self._event = ""
+            self._data = []
+            self._retry = None
+
+            return sse
+
+        if line.startswith(":"):
+            return None
+
+        fieldname, _, value = line.partition(":")
+
+        if value.startswith(" "):
+            value = value[1:]
+
+        if fieldname == "event":
+            self._event = value
+        elif fieldname == "data":
+            self._data.append(value)
+        elif fieldname == "id":
+            if "\0" in value:
+                pass
+            else:
+                self._last_event_id = value
+        elif fieldname == "retry":
+            try:
+                self._retry = int(value)
+            except (TypeError, ValueError):
+                pass
+        else:
+            pass  # Field is ignored.
+
+        return None
+
+
+class EventSource:
+    def __init__(self, response: httpx.Response) -> None:
+        self._response = response
+
+    def _check_content_type(self) -> None:
+        """Check that the response content type is 'text/event-stream'."""
+        self._response.raise_for_status()
+        content_type = self._response.headers.get("content-type", "").partition(";")[0]
+        if "text/event-stream" not in content_type:
+            raise AssertionError(
+                "Expected response header Content-Type to contain 'text/event-stream', "
+                f"got {content_type!r}"
+            )
+
+    @property
+    def response(self) -> httpx.Response:
+        return self._response
+
+    def iter_sse(self) -> Iterator[ServerSentEvent]:
+        self._check_content_type()
+        decoder = SSEDecoder()
+        for line in self._response.iter_lines():
+            line = line.rstrip("\n")
+            sse = decoder.decode(line)
+            if sse is not None:
+                yield sse
+
+    async def aiter_sse(self) -> AsyncIterator[ServerSentEvent]:
+        self._check_content_type()
+        decoder = SSEDecoder()
+        async for line in self._response.aiter_lines():
+            line = line.rstrip("\n")
+            sse = decoder.decode(line)
+            if sse is not None:
+                yield sse
+
+
+@contextmanager
+def connect_sse(
+    client: httpx.Client, method: str, url: str, **kwargs: Any
+) -> Iterator[EventSource]:
+    headers = kwargs.pop("headers", {})
+    headers["Accept"] = "text/event-stream"
+    headers["Cache-Control"] = "no-store"
+
+    with client.stream(method, url, headers=headers, **kwargs) as response:
+        yield EventSource(response)
+
+
+@asynccontextmanager
+async def aconnect_sse(
+    client: httpx.AsyncClient,
+    method: str,
+    url: str,
+    **kwargs: Any,
+) -> AsyncIterator[EventSource]:
+    headers = kwargs.pop("headers", {})
+    headers["Accept"] = "text/event-stream"
+    headers["Cache-Control"] = "no-store"
+
+    async with client.stream(method, url, headers=headers, **kwargs) as response:
+        yield EventSource(response)

langserve/validation.py

@@ -19,16 +19,12 @@ generating OpenAPI specs.
 from typing import Any, Dict, List, Literal, Optional, Sequence, Union
 from uuid import UUID
 
-from langchain.schema import (
-    BaseMessage,
-    ChatGeneration,
-    Document,
-    Generation,
-    RunInfo,
-)
+from langchain_core.documents import Document
+from langchain_core.messages import BaseMessage
+from langchain_core.outputs import ChatGeneration, Generation, RunInfo
 from typing_extensions import Type
 
-from langserve.schema import BatchResponseMetadata, SingletonResponseMetadata
+from langserve.schema import BatchResponseMetadata, InvokeResponseMetadata
 
 try:
     from pydantic.v1 import BaseModel, Field, create_model
@@ -264,19 +260,19 @@ class BatchBaseResponse(BaseModel):
 def create_invoke_response_model(
     namespace: str,
     output_type: Validator,
+    include_callbacks: bool,
 ) -> Type[BaseModel]:
     """Create a pydantic model for the invoke response."""
     # The invoke response uses a key called `output` for the output, so
     # other information can be added to the response at a later date.
-    invoke_response_type = create_model(
-        f"{namespace}InvokeResponse",
-        output=(output_type, Field(..., description="The output of the invocation.")),
-        callback_events=(
-            List[CallbackEvent],
-            Field(..., description="Callback events generated by the server side."),
+
+    fields = {
+        "output": (
+            output_type,
+            Field(..., description="The output of the invocation."),
         ),
-        metadata=(
-            SingletonResponseMetadata,
+        "metadata": (
+            InvokeResponseMetadata,
             Field(
                 ...,
                 description=(
@@ -285,7 +281,21 @@ def create_invoke_response_model(
                 ),
             ),
         ),
+    }
+
+    if include_callbacks:
+        fields["callback_events"] = (
+            List[CallbackEvent],
+            Field(
+                ...,
+                description=("Callback events generated by the server side."),
+            ),
+        )
+
+    invoke_response_type = create_model(
+        f"{namespace}InvokeResponse",
         __base__=InvokeBaseResponse,
+        **fields,
     )
     invoke_response_type.update_forward_refs()
     return invoke_response_type
@@ -294,22 +304,33 @@ def create_invoke_response_model(
 def create_batch_response_model(
     namespace: str,
     output_type: Validator,
+    include_callbacks: bool,
 ) -> Type[BaseModel]:
     """Create a pydantic model for the batch response."""
     # The response uses a key called `output` for the output, so
     # other information can be added to the response at a later date.
-    batch_response_type = create_model(
-        f"{namespace}BatchResponse",
-        output=(
+    fields = {
+        "output": (
             List[output_type],
+            Field(
+                ...,
+                description="The outputs corresponding to the inputs the "
+                "batch request.",
+            ),
+        ),
+        "metadata": (
+            BatchResponseMetadata,
             Field(
                 ...,
                 description=(
-                    "The outputs corresponding to the inputs the batch request."
+                    "Metadata about the response that may be useful to specific clients"
                 ),
             ),
         ),
-        callback_events=(
+    }
+
+    if include_callbacks:
+        fields["callback_events"] = (
             List[List[CallbackEvent]],
             Field(
                 ...,
@@ -319,17 +340,12 @@ def create_batch_response_model(
                     "list corresponds to the callbacks generated for that input."
                 ),
             ),
-        ),
-        metadata=(
-            BatchResponseMetadata,
-            Field(
-                ...,
-                description=(
-                    "Metadata about the response that may be useful to specific clients"
-                ),
-            ),
-        ),
+        )
+
+    batch_response_type = create_model(
+        f"{namespace}BatchResponse",
         __base__=BatchBaseResponse,
+        **fields,
     )
     batch_response_type.update_forward_refs()
     return batch_response_type

pyproject.toml

@@ -1,29 +1,28 @@
 [tool.poetry]
 name = "langserve"
-version = "0.0.48"
+version = "0.2.1"
 description = ""
 readme = "README.md"
 authors = ["LangChain"]
 license = "LangServe"
 repository = "https://github.com/langchain-ai/langserve"
-exclude = ["langserve/playground"]
-include = ["langserve/playground/dist/**/*"]
+exclude = ["langserve/playground,langserve/chat_playground"]
+include = ["langserve/playground/dist/**/*", "langserve/chat_playground/dist/**/*"]
 
 [tool.poetry.dependencies]
 python = "^3.8.1"
 httpx = ">=0.23.0" # May be able to decrease this version
 fastapi = {version = ">=0.90.1,<1", optional = true}
 sse-starlette = {version = "^1.3.0", optional = true}
-httpx-sse = {version = ">=0.3.1", optional = true}
 pydantic = ">=1"
-langchain = ">=0.0.333"
+langchain-core = ">=0.1,<0.3"
 orjson = ">=2"
+pyproject-toml = "^0.0.10"
 
 [tool.poetry.group.dev.dependencies]
 jupyterlab = "^3.6.1"
 fastapi = ">=0.90.1"
 sse-starlette = "^1.3.0"
-httpx-sse = ">=0.3.1"
 
 [tool.poetry.group.typing.dependencies]
 
@@ -45,15 +44,14 @@ openai = "^0.28.0"
 uvicorn = {extras = ["standard"], version = "^0.23.2"}
 fastapi = ">=0.90.1"
 sse-starlette = "^1.3.0"
-httpx-sse = ">=0.3.1"
 
 [tool.poetry.extras]
 # Extras that are used for client
-client = ["httpx-sse"]
+client = []
 # Extras that are used for server
 server = ["sse-starlette", "fastapi"]
 # All
-all = ["httpx-sse", "sse-starlette", "fastapi"]
+all = ["sse-starlette", "fastapi"]
 
 [tool.ruff]
 select = [

tests/unit_tests/test_api_playground.py

@@ -0,0 +1,105 @@
+"""Test the playground API."""
+
+from fastapi import APIRouter, FastAPI
+from httpx import AsyncClient
+from langchain_core.runnables import RunnableLambda
+
+from langserve import add_routes
+
+
+async def test_serve_playground() -> None:
+    """Test the server directly via HTTP requests."""
+    app = FastAPI()
+    add_routes(
+        app,
+        RunnableLambda(lambda foo: "hello"),
+    )
+
+    async with AsyncClient(app=app, base_url="http://localhost:9999") as client:
+        response = await client.get("/playground/index.html")
+        assert response.status_code == 200
+        # Test that we can't access files that do not exist.
+        response = await client.get("/playground/i_do_not_exist.txt")
+        assert response.status_code == 404
+        # Test that we can't access files outside of the playground directory
+        response = await client.get("/playground//etc/passwd")
+        assert response.status_code == 404
+
+
+async def test_serve_playground_with_api_router() -> None:
+    """Test serving playground from an api router with a prefix."""
+    app = FastAPI()
+
+    # Make sure that we can add routers
+    # to an API router
+    router = APIRouter(prefix="/langserve_runnables")
+
+    add_routes(
+        router,
+        RunnableLambda(lambda foo: "hello"),
+        path="/chat",
+    )
+
+    app.include_router(router)
+
+    async with AsyncClient(app=app, base_url="http://localhost:9999") as client:
+        response = await client.get("/langserve_runnables/chat/playground/index.html")
+        assert response.status_code == 200
+
+
+async def test_serve_playground_with_api_router_in_api_router() -> None:
+    """Test serving playground from an api router within an api router."""
+    app = FastAPI()
+
+    router = APIRouter(prefix="/foo")
+
+    add_routes(
+        router,
+        RunnableLambda(lambda foo: "hello"),
+    )
+
+    parent_router = APIRouter(prefix="/parent")
+    parent_router.include_router(router, prefix="/bar")
+
+    # Now add parent router to the app
+    app.include_router(parent_router)
+
+    async with AsyncClient(app=app, base_url="http://localhost:9999") as client:
+        response = await client.get("/parent/bar/foo/playground/index.html")
+        assert response.status_code == 200
+
+
+async def test_root_path_on_playground() -> None:
+    """Test that the playground respects the root_path for requesting assets"""
+
+    for root_path in ("/home/root", "/home/root/"):
+        app = FastAPI(root_path=root_path)
+        add_routes(
+            app,
+            RunnableLambda(lambda foo: "hello"),
+            path="/chat",
+        )
+
+        router = APIRouter(prefix="/router")
+        add_routes(
+            router,
+            RunnableLambda(lambda foo: "hello"),
+            path="/chat",
+        )
+        app.include_router(router)
+
+        async_client = AsyncClient(app=app, base_url="http://localhost:9999")
+
+        response = await async_client.get("/chat/playground/index.html")
+        assert response.status_code == 200
+        assert (
+            f'src="{root_path.rstrip("/")}/chat/playground/assets/'
+            in response.content.decode()
+        ), "html should contain reference to playground assets with root_path prefix"
+
+        response = await async_client.get("/router/chat/playground/index.html")
+        assert response.status_code == 200
+        assert (
+            f'src="{root_path.rstrip("/")}/router/chat/playground/assets/'
+            in response.content.decode()
+        ), "html should contain reference to playground assets with root_path prefix"

tests/unit_tests/test_callbacks.py

@@ -1,14 +1,13 @@
 import uuid
 
+from langchain_core.prompts import ChatPromptTemplate
+
 from langserve.callbacks import AsyncEventAggregatorCallback, replace_uuids
+from tests.unit_tests.utils.llms import FakeListLLM
 
 
 async def test_event_aggregator() -> None:
     """Test that the event aggregator is aggregating events."""
-
-    from langchain.llms import FakeListLLM
-    from langchain.prompts import ChatPromptTemplate
-
     prompt = ChatPromptTemplate.from_template("{question}")
     llm = FakeListLLM(responses=["hello", "world"])
 

tests/unit_tests/test_serialization.py

@@ -4,12 +4,8 @@ from enum import Enum
 from typing import Any
 
 import pytest
-from langchain.schema.messages import (
-    HumanMessage,
-    HumanMessageChunk,
-    SystemMessage,
-)
-from langchain.schema.output import ChatGeneration
+from langchain_core.messages import HumanMessage, HumanMessageChunk, SystemMessage
+from langchain_core.outputs import ChatGeneration
 
 try:
     from pydantic.v1 import BaseModel

tests/unit_tests/test_server_client.py

@@ -2,6 +2,7 @@
 import asyncio
 import datetime
 import json
+import uuid
 from asyncio import AbstractEventLoop
 from contextlib import asynccontextmanager, contextmanager
 from dataclasses import dataclass
@@ -9,6 +10,7 @@ from enum import Enum
 from itertools import cycle
 from typing import (
     Any,
+    AsyncIterator,
     Dict,
     Iterable,
     Iterator,
@@ -26,21 +28,36 @@ import pytest_asyncio
 from fastapi import APIRouter, Depends, FastAPI, Header, HTTPException, Request
 from fastapi.testclient import TestClient
 from httpx import AsyncClient
-from langchain.callbacks.tracers.log_stream import RunLog, RunLogPatch
-from langchain.prompts import PromptTemplate
-from langchain.prompts.base import StringPromptValue
-from langchain.schema.messages import HumanMessage, SystemMessage
-from langchain.schema.runnable import Runnable, RunnableConfig, RunnablePassthrough
-from langchain.schema.runnable.base import RunnableLambda
-from langchain.schema.runnable.utils import ConfigurableField, Input, Output
 from langchain_core.chat_history import BaseChatMessageHistory
 from langchain_core.documents import Document
-from langchain_core.messages import AIMessage, AIMessageChunk, BaseMessage
+from langchain_core.messages import (
+    AIMessage,
+    AIMessageChunk,
+    BaseMessage,
+    HumanMessage,
+    SystemMessage,
+)
 from langchain_core.output_parsers import StrOutputParser
 from langchain_core.outputs import ChatGenerationChunk, LLMResult
-from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
+from langchain_core.prompt_values import StringPromptValue
+from langchain_core.prompts import (
+    ChatPromptTemplate,
+    MessagesPlaceholder,
+    PromptTemplate,
+)
+from langchain_core.runnables import (
+    ConfigurableField,
+    Runnable,
+    RunnableConfig,
+    RunnableLambda,
+    RunnablePassthrough,
+)
 from langchain_core.runnables.history import RunnableWithMessageHistory
+from langchain_core.runnables.utils import Input, Output
+from langchain_core.tracers import RunLog, RunLogPatch
 from langsmith import schemas as ls_schemas
+from langsmith.client import Client
+from langsmith.schemas import FeedbackIngestToken
 from pytest import MonkeyPatch
 from pytest_mock import MockerFixture
 from typing_extensions import Annotated, TypedDict
@@ -54,6 +71,7 @@ from langserve.callbacks import AsyncEventAggregatorCallback
 from langserve.client import RemoteRunnable
 from langserve.lzstring import LZString
 from langserve.schema import CustomUserType
+from tests.unit_tests.utils.stubs import AnyStr
 
 try:
     from pydantic.v1 import BaseModel, Field
@@ -276,73 +294,6 @@ def test_server(app: FastAPI) -> None:
     # assert response.text == "event: data\r\ndata: 2\r\n\r\nevent: end\r\n\r\n"
 
 
-def test_serve_playground(app: FastAPI) -> None:
-    """Test the server directly via HTTP requests."""
-    sync_client = TestClient(app=app)
-    response = sync_client.get("/playground/index.html")
-    assert response.status_code == 200
-    response = sync_client.get("/playground/i_do_not_exist.txt")
-    assert response.status_code == 404
-    response = sync_client.get("/playground//etc/passwd")
-    assert response.status_code == 404
-
-
-async def test_serve_playground_with_api_router() -> None:
-    """Test serving playground from an api router with a prefix."""
-    app = FastAPI()
-
-    # Make sure that we can add routers
-    # to an API router
-    router = APIRouter(prefix="/langserve_runnables")
-
-    add_routes(
-        router,
-        RunnableLambda(lambda foo: "hello"),
-        path="/chat",
-    )
-
-    app.include_router(router)
-    async_client = AsyncClient(app=app, base_url="http://localhost:9999")
-    response = await async_client.get("/langserve_runnables/chat/playground/index.html")
-    assert response.status_code == 200
-
-
-async def test_root_path_on_playground(event_loop: AbstractEventLoop) -> None:
-    """Test that the playground respects the root_path for requesting assets"""
-
-    for root_path in ("/home/root", "/home/root/"):
-        app = FastAPI(root_path=root_path)
-        add_routes(
-            app,
-            RunnableLambda(lambda foo: "hello"),
-            path="/chat",
-        )
-
-        router = APIRouter(prefix="/router")
-        add_routes(
-            router,
-            RunnableLambda(lambda foo: "hello"),
-            path="/chat",
-        )
-        app.include_router(router)
-
-        async_client = AsyncClient(app=app, base_url="http://localhost:9999")
-
-        response = await async_client.get("/chat/playground/index.html")
-        assert response.status_code == 200
-        assert (
-            f'src="{root_path.rstrip("/")}/chat/playground/assets/'
-            in response.content.decode()
-        ), "html should contain reference to playground assets with root_path prefix"
-
-        response = await async_client.get("/router/chat/playground/index.html")
-        assert response.status_code == 200
-        assert (
-            f'src="{root_path.rstrip("/")}/router/chat/playground/assets/'
-            in response.content.decode()
-        ), "html should contain reference to playground assets with root_path prefix"
-
-
 async def test_server_async(app: FastAPI) -> None:
     """Test the server directly via HTTP requests."""
     async with get_async_test_client(app, raise_app_exceptions=True) as async_client:
@@ -430,26 +381,18 @@ async def test_server_async(app: FastAPI) -> None:
     async with get_async_test_client(app, raise_app_exceptions=True) as async_client:
         # Test bad stream requests
         response = await async_client.post("/stream", data="bad json []")
-        stream_events = _decode_eventstream(response.text)
-        assert stream_events[0]["type"] == "error"
-        assert stream_events[0]["data"]["status_code"] == 422
+        assert response.status_code == 422
 
         response = await async_client.post("/stream", json={})
-        stream_events = _decode_eventstream(response.text)
-        assert stream_events[0]["type"] == "error"
-        assert stream_events[0]["data"]["status_code"] == 422
+        assert response.status_code == 422
 
     # test stream_log bad requests
     async with get_async_test_client(app, raise_app_exceptions=True) as async_client:
         response = await async_client.post("/stream_log", data="bad json []")
-        stream_events = _decode_eventstream(response.text)
-        assert stream_events[0]["type"] == "error"
-        assert stream_events[0]["data"]["status_code"] == 422
+        assert response.status_code == 422
 
         response = await async_client.post("/stream_log", json={})
-        stream_events = _decode_eventstream(response.text)
-        assert stream_events[0]["type"] == "error"
-        assert stream_events[0]["data"]["status_code"] == 422
+        assert response.status_code == 422
 
 
 async def test_server_astream_events(app: FastAPI) -> None:
@@ -505,14 +448,10 @@ async def test_server_astream_events(app: FastAPI) -> None:
     # test stream_events with bad requests
     async with get_async_test_client(app, raise_app_exceptions=True) as async_client:
         response = await async_client.post("/stream_events", data="bad json []")
-        stream_events = _decode_eventstream(response.text)
-        assert stream_events[0]["type"] == "error"
-        assert stream_events[0]["data"]["status_code"] == 422
+        assert response.status_code == 422
 
         response = await async_client.post("/stream_events", json={})
-        stream_events = _decode_eventstream(response.text)
-        assert stream_events[0]["type"] == "error"
-        assert stream_events[0]["data"]["status_code"] == 422
+        assert response.status_code == 422
 
 
 async def test_server_bound_async(app_for_config: FastAPI) -> None:
@@ -863,6 +802,31 @@ async def test_astream_log(async_remote_runnable: RemoteRunnable) -> None:
         }
 
 
+async def test_streaming_with_errors() -> None:
+    from langchain_core.runnables import RunnableGenerator
+
+    async def with_errors(inputs: dict) -> AsyncIterator[int]:
+        yield 1
+        raise ValueError("Error")
+        yield 2
+
+    app = FastAPI()
+    add_routes(app, RunnableGenerator(with_errors), path="/with_errors")
+
+    async with get_async_remote_runnable(
+        app, path="/with_errors", raise_app_exceptions=False
+    ) as runnable:
+        chunks = []
+
+        with pytest.raises(httpx.HTTPStatusError) as e:
+            async for chunk in runnable.astream(1):
+                chunks.append(chunk)
+
+        # Check that first chunk was received
+        assert chunks == [1]
+        assert e.value.response.status_code == 500
+
+
 async def test_astream_log_allowlist(event_loop: AbstractEventLoop) -> None:
     """Test async stream with an allowlist."""
 
@@ -2008,10 +1972,14 @@ async def test_per_request_config_modifier_endpoints(
             else:
                 raise ValueError(f"Unknown endpoint {endpoint}")
 
-            if endpoint in {"invoke", "batch"}:
+            if endpoint in {
+                "invoke",
+                "batch",
+                "stream",
+                "stream_log",
+                "astream_events",
+            }:
                 assert response.status_code == 500
-            elif endpoint in {"stream", "stream_log"}:
-                assert '"status_code": 500' in response.text
             else:
                 assert response.status_code != 500
 
@@ -2112,6 +2080,7 @@ async def test_endpoint_configurations() -> None:
         ("GET", "/playground/index.html", {}),
         ("HEAD", "/feedback", {}),
         ("GET", "/feedback", {}),
+        ("POST", "/token_feedback", {}),
         # Check config hashes
         ("POST", "/c/1234/invoke", {"input": 1}),
         ("POST", "/c/1234/batch", {"inputs": [1, 2]}),
@@ -2589,6 +2558,12 @@ async def test_astream_events_with_prompt_model_parser_chain(
                 "name": "GenericFakeChatModel",
                 "tags": ["seq:step:2"],
             },
+            {
+                "data": {"chunk": AIMessageChunk(content="Hello", id=AnyStr())},
+                "event": "on_chat_model_stream",
+                "name": "GenericFakeChatModel",
+                "tags": ["seq:step:2"],
+            },
             {
                 "data": {},
                 "event": "on_parser_start",
@@ -2608,7 +2583,7 @@ async def test_astream_events_with_prompt_model_parser_chain(
                 "tags": [],
             },
             {
-                "data": {"chunk": AIMessageChunk(content="Hello")},
+                "data": {"chunk": AIMessageChunk(content=" ", id=AnyStr())},
                 "event": "on_chat_model_stream",
                 "name": "GenericFakeChatModel",
                 "tags": ["seq:step:2"],
@@ -2626,7 +2601,7 @@ async def test_astream_events_with_prompt_model_parser_chain(
                 "tags": [],
             },
             {
-                "data": {"chunk": AIMessageChunk(content=" ")},
+                "data": {"chunk": AIMessageChunk(content="World!", id=AnyStr())},
                 "event": "on_chat_model_stream",
                 "name": "GenericFakeChatModel",
                 "tags": ["seq:step:2"],
@@ -2643,12 +2618,6 @@ async def test_astream_events_with_prompt_model_parser_chain(
                 "name": "RunnableSequence",
                 "tags": [],
             },
-            {
-                "data": {"chunk": AIMessageChunk(content="World!")},
-                "event": "on_chat_model_stream",
-                "name": "GenericFakeChatModel",
-                "tags": ["seq:step:2"],
-            },
             {
                 "data": {
                     "input": {
@@ -2664,7 +2633,9 @@ async def test_astream_events_with_prompt_model_parser_chain(
                             [
                                 ChatGenerationChunk(
                                     text="Hello World!",
-                                    message=AIMessageChunk(content="Hello World!"),
+                                    message=AIMessageChunk(
+                                        content="Hello World!", id=AnyStr()
+                                    ),
                                 )
                             ]
                         ],
@@ -2678,7 +2649,7 @@ async def test_astream_events_with_prompt_model_parser_chain(
             },
             {
                 "data": {
-                    "input": AIMessageChunk(content="Hello World!"),
+                    "input": AIMessageChunk(content="Hello World!", id=AnyStr()),
                     "output": "Hello World!",
                 },
                 "event": "on_parser_end",
@@ -2816,9 +2787,294 @@ async def test_remote_configurable_remote_runnable() -> None:
         result = await chain_with_history.ainvoke(
             {"question": "hi"}, {"configurable": {"session_id": "1"}}
         )
-        assert result == AIMessage(content="Hello World!")
+        assert result == AIMessage(content="Hello World!", id=AnyStr())
         assert store == {
             "1": InMemoryHistory(
-                messages=[HumanMessage(content="hi"), AIMessage(content="Hello World!")]
+                messages=[
+                    HumanMessage(content="hi"),
+                    AIMessage(content="Hello World!", id=AnyStr()),
+                ]
             )
         }
+
+
+@asynccontextmanager
+async def get_langsmith_client() -> AsyncIterator[MagicMock]:
+    """Get a patched langsmith client."""
+    with patch("langserve.api_handler.ls_client") as mocked_ls_client_package:
+        with patch("langserve.api_handler.tracing_is_enabled") as tracing_is_enabled:
+            tracing_is_enabled.return_value = True
+            mocked_client = MagicMock(auto_spec=Client)
+            mocked_ls_client_package.Client.return_value = mocked_client
+            yield mocked_client
+
+
+async def test_token_feedback_included_in_responses() -> None:
+    """Test that information to leave scoped feedback is passed to the client
+    is present in the server response.
+    """
+    feedback_id = uuid.UUID(int=1)
+    async with get_langsmith_client() as mocked_client:
+        mocked_client.create_presigned_feedback_token.return_value = (
+            FeedbackIngestToken(
+                id=feedback_id,
+                url="feedback_id",
+                expires_at=datetime.datetime(2023, 1, 1),
+            )
+        )
+
+        local_app = FastAPI()
+        add_routes(
+            local_app,
+            RunnableLambda(lambda foo: "hello"),
+            enable_feedback_endpoint=True,
+            token_feedback_config={
+                "key_configs": [
+                    {
+                        "key": "foo",
+                    }
+                ]
+            },
+        )
+
+        async with get_async_test_client(
+            local_app, raise_app_exceptions=True
+        ) as async_client:
+            response = await async_client.post(
+                "/invoke",
+                json={"input": "hello"},
+            )
+
+            json_response = response.json()
+            run_id = json_response["metadata"]["run_id"]
+            assert json_response == {
+                "metadata": {
+                    "feedback_tokens": [
+                        {
+                            "expires_at": "2023-01-01T00:00:00",
+                            "key": "foo",
+                            "token_url": "feedback_id",
+                        }
+                    ],
+                    "run_id": run_id,
+                },
+                "output": "hello",
+            }
+
+            response = await async_client.post(
+                "/batch",
+                json={
+                    "inputs": ["hello", "world"],
+                },
+            )
+            json_response = response.json()
+            responses = json_response["metadata"]["responses"]
+            run_ids = [response["run_id"] for response in responses]
+            assert run_ids == json_response["metadata"]["run_ids"]
+
+            for r in responses:
+                del r["run_id"]
+
+            assert json_response == {
+                "metadata": {
+                    "responses": [
+                        {
+                            "feedback_tokens": [
+                                {
+                                    "expires_at": "2023-01-01T00:00:00",
+                                    "key": "foo",
+                                    "token_url": "feedback_id",
+                                }
+                            ]
+                        },
+                        {
+                            "feedback_tokens": [
+                                {
+                                    "expires_at": "2023-01-01T00:00:00",
+                                    "key": "foo",
+                                    "token_url": "feedback_id",
+                                }
+                            ]
+                        },
+                    ],
+                    "run_ids": run_ids,
+                },
+                "output": ["hello", "hello"],
+            }
+
+            # Test stream
+            response = await async_client.post(
+                "/stream",
+                json={"input": "hello"},
+            )
+            events = _decode_eventstream(response.text)
+            del events[0]["data"]["run_id"]
+            assert events == [
+                {
+                    "data": {
+                        "feedback_tokens": [
+                            {
+                                "expires_at": "2023-01-01T00:00:00",
+                                "key": "foo",
+                                "token_url": "feedback_id",
+                            }
+                        ]
+                    },
+                    "type": "metadata",
+                },
+                {"data": "hello", "type": "data"},
+                {"type": "end"},
+            ]
+
+            # Test astream events
+            response = await async_client.post(
+                "/stream_events",
+                json={"input": "hello"},
+            )
+            events = _decode_eventstream(response.text)
+            for event in events:
+                if "data" in event and "run_id" in event["data"]:
+                    del event["data"]["run_id"]
+
+            # Find the metadata event and pull it out
+            metadata_event = None
+            for event in events:
+                if event["type"] == "metadata":
+                    metadata_event = event
+
+            assert metadata_event == {
+                "data": {
+                    "feedback_tokens": [
+                        {
+                            "expires_at": "2023-01-01T00:00:00",
+                            "key": "foo",
+                            "token_url": "feedback_id",
+                        }
+                    ]
+                },
+                "type": "metadata",
+            }
+
+            # Test astream log
+            response = await async_client.post(
+                "/stream_log",
+                json={"input": "hello"},
+            )
+            events = _decode_eventstream(response.text)
+            for event in events:
+                if "data" in event and "run_id" in event["data"]:
+                    del event["data"]["run_id"]
+
+            # Find the metadata event and pull it out
+            metadata_event = None
+            for event in events:
+                if event["type"] == "metadata":
+                    metadata_event = event
+
+            assert metadata_event == {
+                "data": {
+                    "feedback_tokens": [
+                        {
+                            "expires_at": "2023-01-01T00:00:00",
+                            "key": "foo",
+                            "token_url": "feedback_id",
+                        }
+                    ]
+                },
+                "type": "metadata",
+            }
+
+
+async def test_passing_run_id_from_client() -> None:
+    """test that the client can set a run id if server allows it."""
+    local_app = FastAPI()
+    add_routes(
+        local_app,
+        RunnableLambda(lambda foo: "hello"),
+        config_keys=["run_id"],
+    )
+
+    run_id = uuid.UUID(int=9)
+    run_id2 = uuid.UUID(int=14)
+
+    async with get_async_test_client(
+        local_app, raise_app_exceptions=True
+    ) as async_client:
+        response = await async_client.post(
+            "/invoke",
+            json={"input": "hello", "config": {"run_id": str(run_id)}},
+        )
+        response.raise_for_status()
+        json_response = response.json()
+        assert json_response["metadata"]["run_id"] == str(run_id)
+
+        ## Test batch
+        response = await async_client.post(
+            "/batch",
+            json={
+                "inputs": ["hello", "world"],
+                "config": [{"run_id": str(run_id)}, {"run_id": str(run_id2)}],
+            },
+        )
+        json_response = response.json()
+        responses = json_response["metadata"]["responses"]
+        run_ids = [response["run_id"] for response in responses]
+        assert run_ids == [str(run_id), str(run_id2)]
+
+        # Test stream
+        response = await async_client.post(
+            "/stream",
+            json={"input": "hello", "config": {"run_id": str(run_id)}},
+        )
+        events = _decode_eventstream(response.text)
+        assert events[0]["data"]["run_id"] == str(run_id)
+
+        # Test stream events
+        response = await async_client.post(
+            "/stream_events",
+            json={"input": "hello", "config": {"run_id": str(run_id)}},
+        )
+        events = _decode_eventstream(response.text)
+        assert events[0]["data"]["run_id"] == str(run_id)
+
+
+async def test_passing_bad_runnable_to_add_routes() -> None:
+    """test passing a bad type."""
+    with pytest.raises(TypeError) as e:
+        add_routes(FastAPI(), "not a runnable")
+
+    assert e.match("Expected a Runnable, got <class 'str'>")
+
+
+async def test_token_feedback_endpoint() -> None:
+    """Tests that the feedback endpoint can accept feedback to langsmith."""
+    async with get_langsmith_client() as client:
+        local_app = FastAPI()
+        add_routes(
+            local_app,
+            RunnableLambda(lambda foo: "hello"),
+            token_feedback_config={
+                "key_configs": [
+                    {
+                        "key": "silliness",
+                    }
+                ]
+            },
+        )
+
+        async with get_async_test_client(
+            local_app, raise_app_exceptions=True
+        ) as async_client:
+            response = await async_client.post(
+                "/token_feedback", json={"token_or_url": "some_url", "score": 3}
+            )
+            assert response.status_code == 200
+
+            call = client.create_feedback_from_token.call_args
+            assert call.args[0] == "some_url"
+            assert call.kwargs == {
+                "comment": None,
+                "metadata": {"from_langserve": True},
+                "score": 3,
+                "value": None,
+            }

tests/unit_tests/test_streaming_utils.py

@@ -0,0 +1,40 @@
+"""Test utilities for streaming."""
+import datetime
+import json
+import uuid
+
+from langsmith.schemas import FeedbackIngestToken
+
+from langserve.api_handler import _create_metadata_event
+
+
+def test_create_metadata_event() -> None:
+    """Test that the metadata event is created correctly."""
+    run_id = uuid.UUID(int=7)
+    event = _create_metadata_event(run_id, feedback_ingest_token=None)
+    assert event == {
+        "data": '{"run_id": "00000000-0000-0000-0000-000000000007"}',
+        "event": "metadata",
+    }
+
+    # Test with feedback ingest token
+    feedback_ingest_token = FeedbackIngestToken(
+        id=uuid.UUID(int=8), expires_at=datetime.datetime(2022, 1, 1), url="ingest-url"
+    )
+    event = _create_metadata_event(
+        run_id, feedback_ingest_token=feedback_ingest_token, feedback_key="key"
+    )
+    data = json.loads(event.pop("data"))
+    assert event == {
+        "event": "metadata",
+    }
+    assert data == {
+        "feedback_tokens": [
+            {
+                "expires_at": "2022-01-01T00:00:00",
+                "key": "key",
+                "token_url": "ingest-url",
+            }
+        ],
+        "run_id": "00000000-0000-0000-0000-000000000007",
+    }

tests/unit_tests/test_validation.py

@@ -3,8 +3,8 @@ from unittest.mock import MagicMock
 
 import pytest
 from fastapi import Request
-from langchain.prompts import PromptTemplate
-from langchain.schema.runnable.utils import ConfigurableField
+from langchain_core.prompts import PromptTemplate
+from langchain_core.runnables import ConfigurableField
 
 from langserve.api_handler import _unpack_request_config
 

tests/unit_tests/utils/stubs.py

@@ -0,0 +1,6 @@
+from typing import Any
+
+
+class AnyStr(str):
+    def __eq__(self, other: Any) -> bool:
+        return isinstance(other, str)

tests/unit_tests/utils/test_fake_chat_model.py

@@ -8,6 +8,7 @@ from langchain_core.messages import AIMessage, AIMessageChunk, BaseMessage
 from langchain_core.outputs import ChatGenerationChunk, GenerationChunk
 
 from tests.unit_tests.utils.llms import GenericFakeChatModel
+from tests.unit_tests.utils.stubs import AnyStr
 
 
 def test_generic_fake_chat_model_invoke() -> None:
@@ -15,11 +16,11 @@ def test_generic_fake_chat_model_invoke() -> None:
     infinite_cycle = cycle([AIMessage(content="hello"), AIMessage(content="goodbye")])
     model = GenericFakeChatModel(messages=infinite_cycle)
     response = model.invoke("meow")
-    assert response == AIMessage(content="hello")
+    assert response == AIMessage(content="hello", id=AnyStr())
     response = model.invoke("kitty")
-    assert response == AIMessage(content="goodbye")
+    assert response == AIMessage(content="goodbye", id=AnyStr())
     response = model.invoke("meow")
-    assert response == AIMessage(content="hello")
+    assert response == AIMessage(content="hello", id=AnyStr())
 
 
 async def test_generic_fake_chat_model_ainvoke() -> None:
@@ -27,11 +28,11 @@ async def test_generic_fake_chat_model_ainvoke() -> None:
     infinite_cycle = cycle([AIMessage(content="hello"), AIMessage(content="goodbye")])
     model = GenericFakeChatModel(messages=infinite_cycle)
     response = await model.ainvoke("meow")
-    assert response == AIMessage(content="hello")
+    assert response == AIMessage(content="hello", id=AnyStr())
     response = await model.ainvoke("kitty")
-    assert response == AIMessage(content="goodbye")
+    assert response == AIMessage(content="goodbye", id=AnyStr())
     response = await model.ainvoke("meow")
-    assert response == AIMessage(content="hello")
+    assert response == AIMessage(content="hello", id=AnyStr())
 
 
 async def test_generic_fake_chat_model_stream() -> None:
@@ -44,26 +45,28 @@ async def test_generic_fake_chat_model_stream() -> None:
     model = GenericFakeChatModel(messages=infinite_cycle)
     chunks = [chunk async for chunk in model.astream("meow")]
     assert chunks == [
-        AIMessageChunk(content="hello"),
-        AIMessageChunk(content=" "),
-        AIMessageChunk(content="goodbye"),
+        AIMessageChunk(content="hello", id=AnyStr()),
+        AIMessageChunk(content=" ", id=AnyStr()),
+        AIMessageChunk(content="goodbye", id=AnyStr()),
     ]
 
     chunks = [chunk for chunk in model.stream("meow")]
     assert chunks == [
-        AIMessageChunk(content="hello"),
-        AIMessageChunk(content=" "),
-        AIMessageChunk(content="goodbye"),
+        AIMessageChunk(content="hello", id=AnyStr()),
+        AIMessageChunk(content=" ", id=AnyStr()),
+        AIMessageChunk(content="goodbye", id=AnyStr()),
     ]
 
     # Test streaming of additional kwargs.
     # Relying on insertion order of the additional kwargs dict
-    message = AIMessage(content="", additional_kwargs={"foo": 42, "bar": 24})
+    message = AIMessage(
+        content="", additional_kwargs={"foo": 42, "bar": 24}, id=AnyStr()
+    )
     model = GenericFakeChatModel(messages=cycle([message]))
     chunks = [chunk async for chunk in model.astream("meow")]
     assert chunks == [
-        AIMessageChunk(content="", additional_kwargs={"foo": 42}),
-        AIMessageChunk(content="", additional_kwargs={"bar": 24}),
+        AIMessageChunk(content="", additional_kwargs={"foo": 42}, id=AnyStr()),
+        AIMessageChunk(content="", additional_kwargs={"bar": 24}, id=AnyStr()),
     ]
 
     message = AIMessage(
@@ -81,22 +84,28 @@ async def test_generic_fake_chat_model_stream() -> None:
 
     assert chunks == [
         AIMessageChunk(
-            content="", additional_kwargs={"function_call": {"name": "move_file"}}
+            content="",
+            additional_kwargs={"function_call": {"name": "move_file"}},
+            id=AnyStr(),
         ),
         AIMessageChunk(
             content="",
             additional_kwargs={
                 "function_call": {"arguments": '{\n  "source_path": "foo"'}
             },
+            id=AnyStr(),
         ),
         AIMessageChunk(
-            content="", additional_kwargs={"function_call": {"arguments": ","}}
+            content="",
+            additional_kwargs={"function_call": {"arguments": ","}},
+            id=AnyStr(),
         ),
         AIMessageChunk(
             content="",
             additional_kwargs={
                 "function_call": {"arguments": '\n  "destination_path": "bar"\n}'}
             },
+            id=AnyStr(),
         ),
     ]
 
@@ -116,6 +125,7 @@ async def test_generic_fake_chat_model_stream() -> None:
                 'destination_path": "bar"\n}',
             }
         },
+        id=AnyStr(),
     )
 
 
@@ -128,9 +138,9 @@ async def test_generic_fake_chat_model_astream_log() -> None:
     ]
     final = log_patches[-1]
     assert final.state["streamed_output"] == [
-        AIMessageChunk(content="hello"),
-        AIMessageChunk(content=" "),
-        AIMessageChunk(content="goodbye"),
+        AIMessageChunk(content="hello", id=AnyStr()),
+        AIMessageChunk(content=" ", id=AnyStr()),
+        AIMessageChunk(content="goodbye", id=AnyStr()),
     ]
 
 
@@ -178,8 +188,8 @@ async def test_callback_handlers() -> None:
     # New model
     results = list(model.stream("meow", {"callbacks": [MyCustomAsyncHandler(tokens)]}))
     assert results == [
-        AIMessageChunk(content="hello"),
-        AIMessageChunk(content=" "),
-        AIMessageChunk(content="goodbye"),
+        AIMessageChunk(content="hello", id=AnyStr()),
+        AIMessageChunk(content=" ", id=AnyStr()),
+        AIMessageChunk(content="goodbye", id=AnyStr()),
     ]
     assert tokens == ["hello", " ", "goodbye"]

tests/unit_tests/utils/tracer.py

@@ -1,7 +1,7 @@
 from typing import Dict, List
 from uuid import UUID
 
-from langchain.callbacks.tracers.base import BaseTracer
+from langchain_core.tracers import BaseTracer
 from langsmith.schemas import Run