Release 0.0.44 (#494)

See release notes

.clabot

@@ -1,4 +1,4 @@
 {
-  "contributors": ["eyurtsev", "hwchase17", "nfcampos", "efriis", "jacoblee93", "dqbd", "kreneskyp", "adarsh-jha-dev", "harris", "baskaryan", "hinthornw", "bracesproul", "jakerachleff"],
-  "message": "Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have the username {{usersWithoutCLA}} on file. In order for us to review and merge your code, please complete the Individual Contributor License Agreement here https://forms.gle/Ljhqvt9Gdi1N385W6 .\n\nThis process is done manually on our side, so after signing the form one of the maintainers will add you to the contributors list.\n\nFor more details about why we have a CLA and other contribution guidelines please see: https://github.com/langchain-ai/langserve/blob/main/CONTRIBUTING.md."
+  "contributors": ["eyurtsev", "hwchase17", "nfcampos", "efriis", "jacoblee93", "dqbd", "kreneskyp", "adarsh-jha-dev", "harris", "baskaryan", "hinthornw", "bracesproul", "jakerachleff", "craigsdennis", "anhi", "169", "LarchLiu", "PaulLockett", "RCMatthias", "jwynia", "majiayu000", "mpskex", "shivachittamuru", "sinashaloudegi", "sowsan", "akira", "lucianotonet", "JGalego", "nat-n", "dirien"],
+  "message": "Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have the username {{usersWithoutCLA}} on file. In order for us to review and merge your code, please complete the Individual Contributor License Agreement here https://forms.gle/AQFbtkWRoHXUgipM6 .\n\nThis process is done manually on our side, so after signing the form one of the maintainers will add you to the contributors list.\n\nFor more details about why we have a CLA and other contribution guidelines please see: https://github.com/langchain-ai/langserve/blob/main/CONTRIBUTING.md."
 }

.github/workflows/_pydantic_compatibility.yml

@@ -13,6 +13,7 @@ env:
 
 jobs:
   build:
+    timeout-minutes: 10
     defaults:
       run:
         working-directory: ${{ inputs.working-directory }}

.github/workflows/langserve_ci.yml

@@ -11,6 +11,7 @@ on:
       - '.github/workflows/_test.yml'
       - '.github/workflows/langserve_ci.yml'
       - 'langserve/**'
+      - 'tests/**'
       - 'examples/**'
       - 'pyproject.toml'
       - 'poetry.lock'
@@ -46,6 +47,7 @@ jobs:
       working-directory: .
     secrets: inherit
   test:
+    timeout-minutes: 10
     runs-on: ubuntu-latest
     defaults:
       run:

README.md

@@ -1,51 +1,83 @@
-# LangServe 🦜️🏓
+# 🦜️🏓 LangServe
 
 [![Release Notes](https://img.shields.io/github/release/langchain-ai/langserve)](https://github.com/langchain-ai/langserve/releases)
 [![Downloads](https://static.pepy.tech/badge/langserve/month)](https://pepy.tech/project/langserve)
 [![Open Issues](https://img.shields.io/github/issues-raw/langchain-ai/langserve)](https://github.com/langchain-ai/langserve/issues)
 [![](https://dcbadge.vercel.app/api/server/6adMQxSpJS?compact=true&style=flat)](https://discord.com/channels/1038097195422978059/1170024642245832774)
 
-🚩 We will be releasing a hosted version of LangServe for one-click deployments of LangChain applications. [Sign up here](https://airtable.com/app0hN6sd93QcKubv/shrAjst60xXa6quV2) to get on the waitlist.
+🚩 We will be releasing a hosted version of LangServe for one-click deployments of
+LangChain applications. [Sign up here](https://airtable.com/app0hN6sd93QcKubv/shrAjst60xXa6quV2)
+to get on the waitlist.
 
 ## Overview
 
-`LangServe` helps developers deploy `LangChain` [runnables and chains](https://python.langchain.com/docs/expression_language/) as a REST API.
+[LangServe](https://github.com/langchain-ai/langserve) helps developers
+deploy `LangChain` [runnables and chains](https://python.langchain.com/docs/expression_language/)
+as a REST API.
 
-This library is integrated with [FastAPI](https://fastapi.tiangolo.com/) and uses [pydantic](https://docs.pydantic.dev/latest/) for data validation.
+This library is integrated with [FastAPI](https://fastapi.tiangolo.com/) and
+uses [pydantic](https://docs.pydantic.dev/latest/) for data validation.
 
-In addition, it provides a client that can be used to call into runnables deployed on a server.
-A javascript client is available in [LangChainJS](https://js.langchain.com/docs/api/runnables_remote/classes/RemoteRunnable).
+In addition, it provides a client that can be used to call into runnables deployed on a
+server.
+A JavaScript client is available
+in [LangChain.js](https://js.langchain.com/docs/ecosystem/langserve).
 
 ## Features
 
-- Input and Output schemas automatically inferred from your LangChain object, and enforced on every API call, with rich error messages
+- Input and Output schemas automatically inferred from your LangChain object, and
+  enforced on every API call, with rich error messages
 - API docs page with JSONSchema and Swagger (insert example link)
-- Efficient `/invoke`, `/batch` and `/stream` endpoints with support for many concurrent requests on a single server
-- `/stream_log` endpoint for streaming all (or some) intermediate steps from your chain/agent
-- Playground page at `/playground` with streaming output and intermediate steps
-- Built-in (optional) tracing to [LangSmith](https://www.langchain.com/langsmith), just add your API key (see [Instructions](https://docs.smith.langchain.com/)])
-- All built with battle-tested open-source Python libraries like FastAPI, Pydantic, uvloop and asyncio.
-- Use the client SDK to call a LangServe server as if it was a Runnable running locally (or call the HTTP API directly)
+- Efficient `/invoke/`, `/batch/` and `/stream/` endpoints with support for many
+  concurrent requests on a single server
+- `/stream_log/` endpoint for streaming all (or some) intermediate steps from your
+  chain/agent
+- **new** as of 0.0.40, supports `astream_events` to make it easier to stream without needing to parse the output of `stream_log`.
+- Playground page at `/playground/` with streaming output and intermediate steps
+- Built-in (optional) tracing to [LangSmith](https://www.langchain.com/langsmith), just
+  add your API key (see [Instructions](https://docs.smith.langchain.com/))
+- All built with battle-tested open-source Python libraries like FastAPI, Pydantic,
+  uvloop and asyncio.
+- Use the client SDK to call a LangServe server as if it was a Runnable running
+  locally (or call the HTTP API directly)
 - [LangServe Hub](https://github.com/langchain-ai/langchain/blob/master/templates/README.md)
 
-### Limitations
+## Limitations
 
 - Client callbacks are not yet supported for events that originate on the server
-- OpenAPI docs will not be generated when using Pydantic V2. Fast API does not support [mixing pydantic v1 and v2 namespaces](https://github.com/tiangolo/fastapi/issues/10360). See section below for more details.
+- OpenAPI docs will not be generated when using Pydantic V2. Fast API does not
+  support [mixing pydantic v1 and v2 namespaces](https://github.com/tiangolo/fastapi/issues/10360).
+  See section below for more details.
 
 ## Hosted LangServe
 
-We will be releasing a hosted version of LangServe for one-click deployments of LangChain applications. [Sign up here](https://airtable.com/app0hN6sd93QcKubv/shrAjst60xXa6quV2) to get on the waitlist.
+We will be releasing a hosted version of LangServe for one-click deployments of
+LangChain
+applications. [Sign up here](https://airtable.com/apppQ9p5XuujRl3wJ/shrABpHWdxry8Bacm)
+to get on the waitlist.
 
 ## Security
 
-* Vulnerability in Versions 0.0.13 - 0.0.15 -- playground endpoint allows accessing arbitrary files on server. [Resolved in 0.0.16](https://github.com/langchain-ai/langserve/pull/98).
+* Vulnerability in Versions 0.0.13 - 0.0.15 -- playground endpoint allows accessing
+  arbitrary files on
+  server. [Resolved in 0.0.16](https://github.com/langchain-ai/langserve/pull/98).
+
+## Installation
+
+For both client and server:
+
+```bash
+pip install "langserve[all]"
+```
+
+or `pip install "langserve[client]"` for client code,
+and `pip install "langserve[server]"` for server code.
 
 ## LangChain CLI 🛠️
 
 Use the `LangChain` CLI to bootstrap a `LangServe` project quickly.
 
-To use the langchain CLI make sure that you have a recent version of `langchain-cli` 
+To use the langchain CLI make sure that you have a recent version of `langchain-cli`
 installed. You can install it with `pip install -U langchain-cli`.
 
 ```sh
@@ -54,16 +86,40 @@ langchain app new ../path/to/directory
 
 ## Examples
 
-Get your LangServe instance started quickly with 
+Get your LangServe instance started quickly with
 [LangChain Templates](https://github.com/langchain-ai/langchain/blob/master/templates/README.md).
 
-For more examples, see the templates 
-[index](https://github.com/langchain-ai/langchain/blob/master/templates/docs/INDEX.md) 
-or the [examples](./examples) directory.
+For more examples, see the templates
+[index](https://github.com/langchain-ai/langchain/blob/master/templates/docs/INDEX.md)
+or the [examples](https://github.com/langchain-ai/langserve/tree/main/examples)
+directory.
+
+| Description                                                                                                                                                                                                                                                        | Links                                                                                                                                                                                                                               |
+|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| **LLMs** Minimal example that reserves OpenAI and Anthropic chat models. Uses async, supports batching and streaming.                                                                                                                                              | [server](https://github.com/langchain-ai/langserve/tree/main/examples/llm/server.py), [client](https://github.com/langchain-ai/langserve/blob/main/examples/llm/client.ipynb)                                                       | 
+| **Retriever** Simple server that exposes a retriever as a runnable.                                                                                                                                                                                                | [server](https://github.com/langchain-ai/langserve/tree/main/examples/retrieval/server.py), [client](https://github.com/langchain-ai/langserve/tree/main/examples/retrieval/client.ipynb)                                           | 
+| **Conversational Retriever** A [Conversational Retriever](https://python.langchain.com/docs/expression_language/cookbook/retrieval#conversational-retrieval-chain) exposed via LangServe                                                                           | [server](https://github.com/langchain-ai/langserve/tree/main/examples/conversational_retrieval_chain/server.py), [client](https://github.com/langchain-ai/langserve/tree/main/examples/conversational_retrieval_chain/client.ipynb) | 
+| **Agent** without **conversation history** based on [OpenAI tools](https://python.langchain.com/docs/modules/agents/agent_types/openai_functions_agent)                                                                                             | [server](https://github.com/langchain-ai/langserve/tree/main/examples/agent/server.py), [client](https://github.com/langchain-ai/langserve/tree/main/examples/agent/client.ipynb)                                                   | 
+| **Agent** with **conversation history** based on [OpenAI tools](https://python.langchain.com/docs/modules/agents/agent_types/openai_functions_agent)                                                                                                               | [server](https://github.com/langchain-ai/langserve/blob/main/examples/agent_with_history/server.py), [client](https://github.com/langchain-ai/langserve/blob/main/examples/agent_with_history/client.ipynb)                         | 
+| [RunnableWithMessageHistory](https://python.langchain.com/docs/expression_language/how_to/message_history) to implement chat persisted on backend, keyed off a `session_id` supplied by client.                                                                    | [server](https://github.com/langchain-ai/langserve/tree/main/examples/chat_with_persistence/server.py), [client](https://github.com/langchain-ai/langserve/tree/main/examples/chat_with_persistence/client.ipynb)                   |
+| [RunnableWithMessageHistory](https://python.langchain.com/docs/expression_language/how_to/message_history) to implement chat persisted on backend, keyed off a `conversation_id` supplied by client, and `user_id` (see Auth for implementing `user_id` properly). | [server](https://github.com/langchain-ai/langserve/tree/main/examples/chat_with_persistence_and_user/server.py), [client](https://github.com/langchain-ai/langserve/tree/main/examples/chat_with_persistence_and_user/client.ipynb) | 
+| [Configurable Runnable](https://python.langchain.com/docs/expression_language/how_to/configure) to create a retriever that supports run time configuration of the index name.                                                                                      | [server](https://github.com/langchain-ai/langserve/tree/main/examples/configurable_retrieval/server.py), [client](https://github.com/langchain-ai/langserve/tree/main/examples/configurable_retrieval/client.ipynb)                 | 
+| [Configurable Runnable](https://python.langchain.com/docs/expression_language/how_to/configure) that shows configurable fields and configurable alternatives.                                                                                                      | [server](https://github.com/langchain-ai/langserve/tree/main/examples/configurable_chain/server.py), [client](https://github.com/langchain-ai/langserve/tree/main/examples/configurable_chain/client.ipynb)                         | 
+| **APIHandler** Shows how to use `APIHandler` instead of `add_routes`. This provides more flexibility for developers to define endpoints. Works well with all FastAPI patterns, but takes a bit more effort.                                                        | [server](https://github.com/langchain-ai/langserve/tree/main/examples/api_handler_examples/server.py)                                                                                                                               |
+| **LCEL Example** Example that uses LCEL to manipulate a dictionary input.                                                                                                                                                                                          | [server](https://github.com/langchain-ai/langserve/tree/main/examples/passthrough_dict/client.ipynb), [client](https://github.com/langchain-ai/langserve/tree/main/examples/passthrough_dict/client.ipynb)                          | 
+| **Auth** with `add_routes`: Simple authentication that can be applied across all endpoints associated with app. (Not useful on its own for implementing per user logic.)                                                                                           | [server](https://github.com/langchain-ai/langserve/tree/main/examples/auth/global_deps/server.py)                                                                                                                                   | 
+| **Auth** with `add_routes`: Simple authentication mechanism based on path dependencies. (No useful on its own for implementing per user logic.)                                                                                                                    | [server](https://github.com/langchain-ai/langserve/tree/main/examples/auth/path_dependencies/server.py)                                                                                                                             | 
+| **Auth** with `add_routes`: Implement per user logic and auth for endpoints that use per request config modifier. (**Note**: At the moment, does not integrate with OpenAPI docs.)                                                                                 | [server](https://github.com/langchain-ai/langserve/tree/main/examples/auth/per_req_config_modifier/server.py), [client](https://github.com/langchain-ai/langserve/tree/main/examples/auth/per_req_config_modifier/client.ipynb)     | 
+| **Auth** with `APIHandler`: Implement per user logic and auth that shows how to search only within user owned documents.                                                                                                                                           | [server](https://github.com/langchain-ai/langserve/tree/main/examples/auth/api_handler/server.py), [client](https://github.com/langchain-ai/langserve/tree/main/examples/auth/api_handler/client.ipynb)                             | 
+| **Widgets** Different widgets that can be used with playground (file upload and chat)                                                                                                                                                                              | [server](https://github.com/langchain-ai/langserve/tree/main/examples/widgets/server.py)                                                                                                                                            | 
+| **Widgets** File upload widget used for LangServe playground.                                                                                                                                                                                                      | [server](https://github.com/langchain-ai/langserve/tree/main/examples/file_processing/server.py), [client](https://github.com/langchain-ai/langserve/tree/main/examples/file_processing/client.ipynb)                               | 
+
+## Sample Application
 
 ### Server
 
-Here's a server that deploys an OpenAI chat model, an Anthropic chat model, and a chain that uses
+Here's a server that deploys an OpenAI chat model, an Anthropic chat model, and a chain
+that uses
 the Anthropic model to tell a joke about a topic.
 
 ```python
@@ -73,11 +129,10 @@ from langchain.prompts import ChatPromptTemplate
 from langchain.chat_models import ChatAnthropic, ChatOpenAI
 from langserve import add_routes
 
-
 app = FastAPI(
-  title="LangChain Server",
-  version="1.0",
-  description="A simple api server using Langchain's Runnable interfaces",
+    title="LangChain Server",
+    version="1.0",
+    description="A simple api server using Langchain's Runnable interfaces",
 )
 
 add_routes(
@@ -97,7 +152,7 @@ prompt = ChatPromptTemplate.from_template("tell me a joke about {topic}")
 add_routes(
     app,
     prompt | model,
-    path="/chain",
+    path="/joke",
 )
 
 if __name__ == "__main__":
@@ -110,17 +165,18 @@ if __name__ == "__main__":
 
 If you've deployed the server above, you can view the generated OpenAPI docs using:
 
+> ⚠️ If using pydantic v2, docs will not be generated for *invoke*, *batch*, *stream*,
+*stream_log*. See [Pydantic](#pydantic) section below for more details.
+
 ```sh
 curl localhost:8000/docs
 ```
 
-make sure to **add** the `/docs` suffix. 
+make sure to **add** the `/docs` suffix.
 
-Below will return a 404 until you define a `@app.get("/")`
-
-```sh
-localhost:8000
-```
+> ⚠️ Index page `/` is not defined by **design**, so `curl localhost:8000` or visiting
+> the URL
+> will return a 404. If you want content at `/` define an endpoint `@app.get("/")`.
 
 ### Client
 
@@ -135,7 +191,7 @@ from langserve import RemoteRunnable
 
 openai = RemoteRunnable("http://localhost:8000/openai/")
 anthropic = RemoteRunnable("http://localhost:8000/anthropic/")
-joke_chain = RemoteRunnable("http://localhost:8000/chain/")
+joke_chain = RemoteRunnable("http://localhost:8000/joke/")
 
 joke_chain.invoke({"topic": "parrots"})
 
@@ -161,19 +217,19 @@ chain = prompt | RunnableMap({
     "anthropic": anthropic,
 })
 
-chain.batch([{ "topic": "parrots" }, { "topic": "cats" }])
+chain.batch([{"topic": "parrots"}, {"topic": "cats"}])
 ```
 
 In TypeScript (requires LangChain.js version 0.0.166 or later):
 
 ```typescript
-import { RemoteRunnable } from "langchain/runnables/remote";
+import { RemoteRunnable } from "@langchain/core/runnables/remote";
 
 const chain = new RemoteRunnable({
-  url: `http://localhost:8000/chain/invoke/`,
+    url: `http://localhost:8000/joke/`,
 });
 const result = await chain.invoke({
-  topic: "cats",
+    topic: "cats",
 });
 ```
 
@@ -181,8 +237,9 @@ Python using `requests`:
 
 ```python
 import requests
+
 response = requests.post(
-    "http://localhost:8000/chain/invoke/",
+    "http://localhost:8000/joke/invoke",
     json={'input': {'topic': 'cats'}}
 )
 response.json()
@@ -191,7 +248,7 @@ response.json()
 You can also use `curl`:
 
 ```sh
-curl --location --request POST 'http://localhost:8000/chain/invoke/' \
+curl --location --request POST 'http://localhost:8000/joke/invoke' \
     --header 'Content-Type: application/json' \
     --data-raw '{
         "input": {
@@ -207,9 +264,9 @@ The following code:
 ```python
 ...
 add_routes(
-  app,
-  runnable,
-  path="/my_runnable",
+    app,
+    runnable,
+    path="/my_runnable",
 )
 ```
 
@@ -218,42 +275,78 @@ adds of these endpoints to the server:
 - `POST /my_runnable/invoke` - invoke the runnable on a single input
 - `POST /my_runnable/batch` - invoke the runnable on a batch of inputs
 - `POST /my_runnable/stream` - invoke on a single input and stream the output
-- `POST /my_runnable/stream_log` - invoke on a single input and stream the output, including output of intermediate steps as it's generated
+- `POST /my_runnable/stream_log` - invoke on a single input and stream the output,
+  including output of intermediate steps as it's generated
+- `POST /my_runnable/astream_events` - invoke on a single input and stream events as they are generated,
+ including from intermediate steps.
 - `GET /my_runnable/input_schema` - json schema for input to the runnable
 - `GET /my_runnable/output_schema` - json schema for output of the runnable
 - `GET /my_runnable/config_schema` - json schema for config of the runnable
 
+These endpoints match
+the [LangChain Expression Language interface](https://python.langchain.com/docs/expression_language/interface) --
+please reference this documentation for more details.
+
 ## Playground
 
-You can find a playground page for your runnable at `/my_runnable/playground`. This exposes a simple UI to [configure](https://python.langchain.com/docs/expression_language/how_to/configure) and invoke your runnable with streaming output and intermediate steps.
+You can find a playground page for your runnable at `/my_runnable/playground/`. This
+exposes a simple UI
+to [configure](https://python.langchain.com/docs/expression_language/how_to/configure)
+and invoke your runnable with streaming output and intermediate steps.
 
-![image](https://github.com/langchain-ai/langserve/assets/3205522/5ca56e29-f1bb-40f4-84b5-15916384a276)
+<p align="center">
+<img src="https://github.com/langchain-ai/langserve/assets/3205522/5ca56e29-f1bb-40f4-84b5-15916384a276" width="50%"/>
+</p>
 
-## Installation
+### Widgets
 
-For both client and server:
+The playground supports [widgets](#playground-widgets) and can be used to test your
+runnable with different inputs. See the [widgets](#widgets) section below for more
+details.
 
-```bash
-pip install "langserve[all]"
-```
+### Sharing
 
-or `pip install "langserve[client]"` for client code, and `pip install "langserve[server]"` for server code.
+In addition, for configurable runnables, the playground will allow you to configure the
+runnable and share a link with the configuration:
+
+<p align="center">
+<img src="https://github.com/langchain-ai/langserve/assets/3205522/86ce9c59-f8e4-4d08-9fa3-62030e0f521d" width="50%"/>
+</p>
 
 ## Legacy Chains
 
-LangServe works with both Runnables (constructed via [LangChain Expression Language](https://python.langchain.com/docs/expression_language/)) and legacy chains (inheriting from `Chain`).
-However, some of the input schemas for legacy chains may be incomplete/incorrect, leading to errors.
+LangServe works with both Runnables (constructed
+via [LangChain Expression Language](https://python.langchain.com/docs/expression_language/))
+and legacy chains (inheriting from `Chain`).
+However, some of the input schemas for legacy chains may be incomplete/incorrect,
+leading to errors.
 This can be fixed by updating the `input_schema` property of those chains in LangChain.
-If you encounter any errors, please open an issue on THIS repo, and we will work to address it.
-
-## Handling Authentication
-
-If you need to add authentication to your server,
-please reference FastAPI's [security documentation](https://fastapi.tiangolo.com/tutorial/security/)
-and [middleware documentation](https://fastapi.tiangolo.com/tutorial/middleware/).
+If you encounter any errors, please open an issue on THIS repo, and we will work to
+address it.
 
 ## Deployment
 
+### Deploy to AWS
+
+You can deploy to AWS using the [AWS Copilot CLI](https://aws.github.io/copilot-cli/)
+
+```bash
+copilot init --app [application-name] --name [service-name] --type 'Load Balanced Web Service' --dockerfile './Dockerfile' --deploy
+```
+
+Click [here](https://aws.amazon.com/containers/copilot/) to learn more.
+
+### Deploy to Azure
+
+You can deploy to Azure using Azure Container Apps (Serverless):
+
+```
+az containerapp up --name [container-app-name] --source . --resource-group [resource-group-name] --environment  [environment-name] --ingress external --target-port 8001 --env-vars=OPENAI_API_KEY=your_key
+```
+
+You can find more
+info [here](https://learn.microsoft.com/en-us/azure/container-apps/containerapp-up)
+
 ### Deploy to GCP
 
 You can deploy to GCP Cloud Run using the following command:
@@ -262,33 +355,119 @@ You can deploy to GCP Cloud Run using the following command:
 gcloud run deploy [your-service-name] --source . --port 8001 --allow-unauthenticated --region us-central1 --set-env-vars=OPENAI_API_KEY=your_key
 ```
 
+### Deploy using Infrastructure as Code
+
+#### Pulumi
+
+You can deploy your LangServe server with [Pulumi](https://www.pulumi.com/) using your preferred general purpose language. Below are some quickstart 
+examples for deploying LangServe to different cloud providers.
+
+These examples are a good starting point for your own infrastructure as code (IaC) projects. You can easily modify them to suit your needs.
+
+
+| Cloud | Language   | Repository                                          | Quickstart                                                                                                                                  |
+|-------|------------|-----------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------|
+| AWS   | dotnet     | https://github.com/pulumi/examples/aws-cs-langserve | [![Deploy](https://get.pulumi.com/new/button.svg)](https://app.pulumi.com/new?template=https://github.com/pulumi/examples/aws-cs-langserve) |
+| AWS   | golang     | https://github.com/pulumi/examples/aws-go-langserve | [![Deploy](https://get.pulumi.com/new/button.svg)](https://app.pulumi.com/new?template=https://github.com/pulumi/examples/aws-go-langserve) |
+| AWS   | python     | https://github.com/pulumi/examples/aws-py-langserve | [![Deploy](https://get.pulumi.com/new/button.svg)](https://app.pulumi.com/new?template=https://github.com/pulumi/examples/aws-py-langserve) |
+| AWS   | typescript | https://github.com/pulumi/examples/aws-ts-langserve | [![Deploy](https://get.pulumi.com/new/button.svg)](https://app.pulumi.com/new?template=https://github.com/pulumi/examples/aws-ts-langserve) |
+| AWS   | javascript | https://github.com/pulumi/examples/aws-js-langserve | [![Deploy](https://get.pulumi.com/new/button.svg)](https://app.pulumi.com/new?template=https://github.com/pulumi/examples/aws-js-langserve) |
+
+
+
+
+### Community Contributed
+
+#### Deploy to Railway
+
+[Example Railway Repo](https://github.com/PaulLockett/LangServe-Railway/tree/main)
+
+[![Deploy on Railway](https://railway.app/button.svg)](https://railway.app/template/pW9tXP?referralCode=c-aq4K)
+
 ## Pydantic
 
 LangServe provides support for Pydantic 2 with some limitations.
 
-1. OpenAPI docs will not be generated for invoke/batch/stream/stream_log when using Pydantic V2. Fast API does not support [mixing pydantic v1 and v2 namespaces].
-2. LangChain uses the v1 namespace in Pydantic v2. Please read the [following guidelines to ensure compatibility with LangChain](https://github.com/langchain-ai/langchain/discussions/9337)
+1. OpenAPI docs will not be generated for invoke/batch/stream/stream_log when using
+   Pydantic V2. Fast API does not support [mixing pydantic v1 and v2 namespaces].
+2. LangChain uses the v1 namespace in Pydantic v2. Please read
+   the [following guidelines to ensure compatibility with LangChain](https://github.com/langchain-ai/langchain/discussions/9337)
 
-Except for these limitations, we expect the API endpoints, the playground and any other features to work as expected. 
+Except for these limitations, we expect the API endpoints, the playground and any other
+features to work as expected.
 
 ## Advanced
 
+### Handling Authentication
+
+If you need to add authentication to your server, please read Fast API's documentation
+about [dependencies](https://fastapi.tiangolo.com/tutorial/dependencies/)
+and [security](https://fastapi.tiangolo.com/tutorial/security/).
+
+The below examples show how to wire up authentication logic LangServe endpoints using FastAPI primitives.
+
+You are responsible for providing the actual authentication logic, the users table etc. 
+
+If you're not sure what you're doing, you could try using an existing solution [Auth0](https://auth0.com/).
+
+#### Using add_routes
+
+If you're using `add_routes`, see
+examples [here](https://github.com/langchain-ai/langserve/tree/main/examples/auth).
+
+| Description                                                                                                                                                                                                                                                        | Links                                                                                                                                                                                                                                |
+|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| **Auth** with `add_routes`: Simple authentication that can be applied across all endpoints associated with app. (Not useful on its own for implementing per user logic.)                                                                                           | [server](https://github.com/langchain-ai/langserve/tree/main/examples/auth/global_deps/server.py)                                                                                                                                    | 
+| **Auth** with `add_routes`: Simple authentication mechanism based on path dependencies. (No useful on its own for implementing per user logic.)                                                                                                                    | [server](https://github.com/langchain-ai/langserve/tree/main/examples/auth/path_dependencies/server.py)                                                                                                                              | 
+| **Auth** with `add_routes`: Implement per user logic and auth for endpoints that use per request config modifier. (**Note**: At the moment, does not integrate with OpenAPI docs.)                                                                                 | [server](https://github.com/langchain-ai/langserve/tree/main/examples/auth/per_req_config_modifier/server.py), [client](https://github.com/langchain-ai/langserve/tree/main/examples/auth/per_req_config_modifier/client.ipynb)     | 
+
+Alternatively, you can use FastAPI's [middleware](https://fastapi.tiangolo.com/tutorial/middleware/).
+
+Using global dependencies and path dependencies has the advantage that auth will be properly supported in the OpenAPI docs page, but
+these are not sufficient for implement per user logic (e.g., making an application that can search only within user owned documents).
+
+If you need to implement per user logic, you can use the `per_req_config_modifier` or `APIHandler` (below) to implement this logic.
+
+**Per User**
+
+If you need authorization or logic that is user dependent,
+specify `per_req_config_modifier` when using `add_routes`. Use a callable receives the
+raw `Request` object and can extract relevant information from it for authentication and
+authorization purposes.
+
+#### Using APIHandler
+
+If you feel comfortable with FastAPI and python, you can use LangServe's [APIHandler](https://github.com/langchain-ai/langserve/blob/main/examples/api_handler_examples/server.py).
+
+| Description                                                                                                                                                                                                                                                        | Links                                                                                                                                                                                                                                |
+|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| **Auth** with `APIHandler`: Implement per user logic and auth that shows how to search only within user owned documents.                                                                                                                                           | [server](https://github.com/langchain-ai/langserve/tree/main/examples/auth/api_handler/server.py), [client](https://github.com/langchain-ai/langserve/tree/main/examples/auth/api_handler/client.ipynb)                             | 
+| **APIHandler** Shows how to use `APIHandler` instead of `add_routes`. This provides more flexibility for developers to define endpoints. Works well with all FastAPI patterns, but takes a bit more effort.                                                        | [server](https://github.com/langchain-ai/langserve/tree/main/examples/api_handler_examples/server.py), [client](https://github.com/langchain-ai/langserve/tree/main/examples/api_handler_examples/client.ipynb)                     |
+
+It's a bit more work, but gives you complete control over the endpoint definitions, so
+you can do whatever custom logic you need for auth.
+
 ### Files
 
 LLM applications often deal with files. There are different architectures
 that can be made to implement file processing; at a high level:
 
-1. The file may be uploaded to the server via a dedicated endpoint and processed using a separate endpoint
-2. The file may be uploaded by either value (bytes of file) or reference (e.g., s3 url to file content)
+1. The file may be uploaded to the server via a dedicated endpoint and processed using a
+   separate endpoint
+2. The file may be uploaded by either value (bytes of file) or reference (e.g., s3 url
+   to file content)
 3. The processing endpoint may be blocking or non-blocking
-4. If significant processing is required, the processing may be offloaded to a dedicated process pool
+4. If significant processing is required, the processing may be offloaded to a dedicated
+   process pool
 
 You should determine what is the appropriate architecture for your application.
 
-Currently, to upload files by value to a runnable, use base64 encoding for the 
-file (`multipart/form-data` is not supported yet). 
+Currently, to upload files by value to a runnable, use base64 encoding for the
+file (`multipart/form-data` is not supported yet).
 
-Here's an [example](https://github.com/langchain-ai/langserve/tree/main/examples/file_processing) that shows
+Here's
+an [example](https://github.com/langchain-ai/langserve/tree/main/examples/file_processing)
+that shows
 how to use base64 encoding to send a file to a remote runnable.
 
 Remember, you can always upload files by reference (e.g., s3 url) or upload them as
@@ -321,7 +500,7 @@ def func(x: Any) -> int:
 
 
 runnable = RunnableLambda(func).with_types(
-    input_schema=int,
+    input_type=int,
 )
 
 add_routes(app, runnable)
@@ -329,7 +508,7 @@ add_routes(app, runnable)
 
 ### Custom User Types
 
-Inherit from `CustomUserType` if you want the data to de-serialize into a 
+Inherit from `CustomUserType` if you want the data to de-serialize into a
 pydantic model rather than the equivalent dict representation.
 
 At the moment, this type only works *server* side and is used
@@ -356,12 +535,13 @@ def func(foo: Foo) -> int:
     assert isinstance(foo, Foo)
     return foo.bar
 
+
 # Note that the input and output type are automatically inferred!
 # You do not need to specify them.
 # runnable = RunnableLambda(func).with_types( # <-- Not needed in this case
-#     input_schema=Foo,
-#     output_schema=int,
-# 
+#     input_type=Foo,
+#     output_type=int,
+#
 add_routes(app, RunnableLambda(func), path="/foo")
 ```
 
@@ -369,11 +549,20 @@ add_routes(app, RunnableLambda(func), path="/foo")
 
 The playground allows you to define custom widgets for your runnable from the backend.
 
-- A widget is specified at the field level and shipped as part of the JSON schema of the input type
-- A widget must contain a key called `type` with the value being one of a well known list of widgets
-- Other widget keys will be associated with values that describe paths in a JSON object
+Here are a few examples:
 
-General schema:
+| Description                                                                                                                                                                                                                                                        | Links                                                                                                                                                                                                                                |
+|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| **Widgets** Different widgets that can be used with playground (file upload and chat)                                                                                                                                                                              | [server](https://github.com/langchain-ai/langserve/tree/main/examples/widgets/server.py), [client](https://github.com/langchain-ai/langserve/tree/main/examples/widgets/client.ipynb)                                               | 
+| **Widgets** File upload widget used for LangServe playground.                                                                                                                                                                                                      | [server](https://github.com/langchain-ai/langserve/tree/main/examples/file_processing/server.py), [client](https://github.com/langchain-ai/langserve/tree/main/examples/file_processing/client.ipynb)                               | 
+
+#### Schema
+
+- A widget is specified at the field level and shipped as part of the JSON schema of the
+  input type
+- A widget must contain a key called `type` with the value being one of a well known
+  list of widgets
+- Other widget keys will be associated with values that describe paths in a JSON object
 
 ```typescript
 type JsonPath = number | string | (number | string)[];
@@ -386,11 +575,24 @@ type Widget = {
 };
 ```
 
+### Available Widgets
+
+There are only two widgets that the user can specify manually right now:
+
+1. File Upload Widget
+2. Chat History Widget
+
+See below more information about these widgets.
+
+All other widgets on the playground UI are created and managed automatically by the UI
+based on the config schema of the Runnable. When you create Configurable Runnables,
+the playground should create appropriate widgets for you to control the behavior.
 
 #### File Upload Widget
 
 Allows creation of a file upload input in the UI playground for files
-that are uploaded as base64 encoded strings. Here's the full [example](https://github.com/langchain-ai/langserve/tree/main/examples/file_processing).
+that are uploaded as base64 encoded strings. Here's the
+full [example](https://github.com/langchain-ai/langserve/tree/main/examples/file_processing).
 
 Snippet:
 
@@ -413,3 +615,83 @@ class FileProcessingRequest(CustomUserType):
     num_chars: int = 100
 
 ```
+
+Example widget:
+
+<p align="center">
+<img src="https://github.com/langchain-ai/langserve/assets/3205522/52199e46-9464-4c2e-8be8-222250e08c3f" width="50%"/>
+</p>
+
+### Chat Widget
+
+Look
+at [widget example](https://github.com/langchain-ai/langserve/tree/main/examples/widgets/server.py).
+
+To define a chat widget, make sure that you pass "type": "chat".
+
+* "input" is JSONPath to the field in the *Request* that has the new input message.
+* "output" is JSONPath to the field in the *Response* that has new output message(s).
+* Don't specify these fields if the entire input or output should be used as they are (
+  e.g., if the output is a list of chat messages.)
+
+Here's a snippet:
+
+```python
+
+class ChatHistory(CustomUserType):
+    chat_history: List[Tuple[str, str]] = Field(
+        ...,
+        examples=[[("human input", "ai response")]],
+        extra={"widget": {"type": "chat", "input": "question", "output": "answer"}},
+    )
+    question: str
+
+
+def _format_to_messages(input: ChatHistory) -> List[BaseMessage]:
+    """Format the input to a list of messages."""
+    history = input.chat_history
+    user_input = input.question
+
+    messages = []
+
+    for human, ai in history:
+        messages.append(HumanMessage(content=human))
+        messages.append(AIMessage(content=ai))
+    messages.append(HumanMessage(content=user_input))
+    return messages
+
+
+model = ChatOpenAI()
+chat_model = RunnableParallel({"answer": (RunnableLambda(_format_to_messages) | model)})
+add_routes(
+    app,
+    chat_model.with_types(input_type=ChatHistory),
+    config_keys=["configurable"],
+    path="/chat",
+)
+```
+
+Example widget:
+<p align="center">
+<img src="https://github.com/langchain-ai/langserve/assets/3205522/a71ff37b-a6a9-4857-a376-cf27c41d3ca4" width="50%"/>
+</p>
+
+### Enabling / Disabling Endpoints (LangServe >=0.0.33)
+
+You can enable / disable which endpoints are exposed when adding routes for a given chain.
+
+Use `enabled_endpoints` if you want to make sure to never get a new endpoint when upgrading langserve to a newer
+verison.
+
+Enable: The code below will only enable `invoke`, `batch` and the
+corresponding `config_hash` endpoint variants.
+
+```python
+add_routes(app, chain, enabled_endpoints=["invoke", "batch", "config_hashes"], path="/mychain")
+```
+
+Disable: The code below will disable the playground for the chain
+
+```python
+add_routes(app, chain, disabled_endpoints=["playground"], path="/mychain")
+```

examples/agent/client.ipynb

@@ -27,7 +27,8 @@
      "data": {
       "text/plain": [
        "{'output': {'output': 'Eugene thinks that cats like fish.'},\n",
-       " 'callback_events': []}"
+       " 'callback_events': [],\n",
+       " 'metadata': {'run_id': 'f16d95e5-dd8f-48d1-8668-4b33a54023fb'}}"
       ]
      },
      "execution_count": 1,
@@ -95,7 +96,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
+   "execution_count": 4,
    "metadata": {
     "tags": []
    },
@@ -106,7 +107,7 @@
        "{'output': 'Eugene thinks that cats like fish.'}"
       ]
      },
-     "execution_count": 6,
+     "execution_count": 4,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -114,6 +115,816 @@
    "source": [
     "remote_runnable.invoke({\"input\": \"what does eugene think of cats?\"})"
    ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Stream\n",
+    "\n",
+    "Please note that streaming alternates between actions and observations. It does not stream individual tokens! If you need to stream individual tokens you will need to use astream_log!"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "--\n",
+      "{'actions': [AgentActionMessageLog(tool='get_eugene_thoughts', tool_input={'query': 'cats'}, log=\"\\nInvoking: `get_eugene_thoughts` with `{'query': 'cats'}`\\n\\n\\n\", message_log=[AIMessage(content='', additional_kwargs={'function_call': {'name': 'get_eugene_thoughts', 'arguments': '{\\n  \"query\": \"cats\"\\n}'}})])], 'messages': [AIMessage(content='', additional_kwargs={'function_call': {'name': 'get_eugene_thoughts', 'arguments': '{\\n  \"query\": \"cats\"\\n}'}})]}\n",
+      "--\n",
+      "{'steps': [{'action': AgentActionMessageLog(tool='get_eugene_thoughts', tool_input={'query': 'cats'}, log=\"\\nInvoking: `get_eugene_thoughts` with `{'query': 'cats'}`\\n\\n\\n\", message_log=[AIMessage(content='', additional_kwargs={'function_call': {'name': 'get_eugene_thoughts', 'arguments': '{\\n  \"query\": \"cats\"\\n}'}})]), 'observation': [Document(page_content='cats like fish'), Document(page_content='dogs like sticks')]}], 'messages': [FunctionMessage(content=\"[Document(page_content='cats like fish'), Document(page_content='dogs like sticks')]\", name='get_eugene_thoughts')]}\n",
+      "--\n",
+      "{'output': \"Eugene thinks that cats like fish. Now let me tell you a story about that thought.\\n\\nOnce upon a time, in a small village, there lived a mischievous cat named Whiskers. Whiskers was known for his love for fish. Every day, he would venture out to the nearby river in search of his favorite food.\\n\\nOne sunny day, Whiskers set out on his usual fish-hunting expedition. As he approached the riverbank, he noticed a group of fishermen casting their nets into the water. Whiskers couldn't resist the temptation and decided to join in on the action.\\n\\nWith his agile paws, Whiskers skillfully maneuvered through the fishermen, strategically positioning himself to snatch the fish caught in their nets. The fishermen were amazed by Whiskers' quick reflexes and couldn't help but laugh at the sight of a cat fishing alongside them.\\n\\nWhiskers continued his fish-stealing escapades for several days, becoming somewhat of a local legend. People from neighboring villages would come to witness the incredible sight of a cat outsmarting seasoned fishermen.\\n\\nOne day, as Whiskers was enjoying his stolen fish by the river, he noticed a stray dog named Rover approaching him. Rover had heard about Whiskers' fishing talents and was intrigued by the cat's abilities.\\n\\nCurious, Whiskers decided to share his secret with Rover. He taught him the art of stealth and how to snatch fish from the nets without being noticed. Rover, being a quick learner, soon became Whiskers' partner in crime.\\n\\nTogether, Whiskers and Rover formed an unbeatable duo, leaving the fishermen puzzled and amazed at the disappearing fish. The villagers, entertained by their antics, started leaving fish out for Whiskers and Rover as a token of appreciation.\\n\\nAnd so, Whiskers and Rover continued their fish-stealing adventures, bringing joy and laughter to the village. Eugene's thought about cats and their love for fish certainly came to life in the mischievous and clever Whiskers, who proved that cats truly have a special affinity for fish.\", 'messages': [AIMessage(content=\"Eugene thinks that cats like fish. Now let me tell you a story about that thought.\\n\\nOnce upon a time, in a small village, there lived a mischievous cat named Whiskers. Whiskers was known for his love for fish. Every day, he would venture out to the nearby river in search of his favorite food.\\n\\nOne sunny day, Whiskers set out on his usual fish-hunting expedition. As he approached the riverbank, he noticed a group of fishermen casting their nets into the water. Whiskers couldn't resist the temptation and decided to join in on the action.\\n\\nWith his agile paws, Whiskers skillfully maneuvered through the fishermen, strategically positioning himself to snatch the fish caught in their nets. The fishermen were amazed by Whiskers' quick reflexes and couldn't help but laugh at the sight of a cat fishing alongside them.\\n\\nWhiskers continued his fish-stealing escapades for several days, becoming somewhat of a local legend. People from neighboring villages would come to witness the incredible sight of a cat outsmarting seasoned fishermen.\\n\\nOne day, as Whiskers was enjoying his stolen fish by the river, he noticed a stray dog named Rover approaching him. Rover had heard about Whiskers' fishing talents and was intrigued by the cat's abilities.\\n\\nCurious, Whiskers decided to share his secret with Rover. He taught him the art of stealth and how to snatch fish from the nets without being noticed. Rover, being a quick learner, soon became Whiskers' partner in crime.\\n\\nTogether, Whiskers and Rover formed an unbeatable duo, leaving the fishermen puzzled and amazed at the disappearing fish. The villagers, entertained by their antics, started leaving fish out for Whiskers and Rover as a token of appreciation.\\n\\nAnd so, Whiskers and Rover continued their fish-stealing adventures, bringing joy and laughter to the village. Eugene's thought about cats and their love for fish certainly came to life in the mischievous and clever Whiskers, who proved that cats truly have a special affinity for fish.\")]}\n"
+     ]
+    }
+   ],
+   "source": [
+    "async for chunk in remote_runnable.astream({\"input\": \"what does eugene think of cats? Then tell me a story about that thought.\"}):\n",
+    "    print('--')\n",
+    "    print(chunk)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Stream Events\n",
+    "\n",
+    "The client is looking for a runnable name called `agent` for the chain events. This name was defined on the server side using `runnable.with_config({\"run_name\": \"agent\"}`"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Starting agent: agent with input: {'input': 'what does eugene think of cats? Then tell me a story about that thought.'}\n",
+      "--\n",
+      "Starting tool: get_eugene_thoughts with inputs: {'query': 'cats'}\n",
+      "Done tool: get_eugene_thoughts\n",
+      "Tool output was: [Document(page_content='cats like fish'), Document(page_content='dogs like sticks')]\n",
+      "--\n",
+      "E|ug|ene| thinks| that| cats| like| fish|.| Now| let| me| tell| you| a| story| about| that| thought|:\n",
+      "\n",
+      "|Once| upon| a| time|,| in| a| small| village|,| there| lived| a| curious| cat| named| Wh|isk|ers|.| Wh|isk|ers| was| known| for| his| love| of| fish|.| Every| day|,| he| would| venture| to| the| nearby| river| in| search| of| his| favorite| meal|.\n",
+      "\n",
+      "|One| sunny| morning|,| Wh|isk|ers| set| out| on| his| daily| fish|-h|unting| expedition|.| As| he| approached| the| river|,| he| could| smell| the| fresh| scent| of| the| water| and| feel| the| excitement| building| up| inside| him|.| Wh|isk|ers| knew| that| today| might| be| his| lucky| day|.\n",
+      "\n",
+      "|He| carefully| ti|pto|ed| along| the| river|bank|,| his| eyes| fixed| on| the| water|'s| surface|.| Suddenly|,| he| spotted| a| shimmer|ing| fish| swimming| gracefully| through| the| clear| blue| water|.| Wh|isk|ers| c|rou|ched| low|,| ready| to| p|ounce|.\n",
+      "\n",
+      "|With| lightning| speed|,| he| le|aped| into| the| air|,| his| p|aws| out|st|retched| towards| the| fish|.| Splash|!| Wh|isk|ers| landed| right| in| the| middle| of| the| river|,| causing| r|ipples| to| spread| in| all| directions|.| But| he| didn|'t| care|.| All| he| wanted| was| that| delicious| fish|.\n",
+      "\n",
+      "|Wh|isk|ers| chased| the| fish| with| all| his| might|,| dart|ing| through| the| water| with| elegance| and| precision|.| The| fish| sw|am| gracefully|,| trying| to| escape| Wh|isk|ers|'| determined| pursuit|.| But| the| cat| was| relentless|.\n",
+      "\n",
+      "|After| a| few| moments| of| intense| chase|,| Wh|isk|ers| finally| managed| to| catch| the| fish| in| his| p|aws|.| He| triumph|antly| carried| it| to| the| river|bank|,| where| he| enjoyed| his| well|-des|erved| feast|.| The| taste| of| the| fresh| fish| was| heavenly|,| satisfying| his| hunger| and| bringing| a| content|ed| smile| to| his| face|.\n",
+      "\n",
+      "|From| that| day| on|,| Wh|isk|ers| became| known| as| the| legendary| fish|-catching| cat| in| the| village|.| People| would| often| gather| by| the| river| to| watch| him| in| action|,| amazed| by| his| agility| and| determination|.| Wh|isk|ers| taught| everyone| the| importance| of| perseverance| and| following| one|'s| passion|,| just| like| he| pursued| his| love| for| fish|.\n",
+      "\n",
+      "|And| so|,| Wh|isk|ers| continued| his| fish|-h|unting| adventures|,| spreading| joy| and| inspiration| to| everyone| he| encountered|.| He| proved| that| when| you| have| a| passion| for| something|,| nothing| can| stop| you| from| achieving| it| –| just| like| cats| and| their| love| for| fish|.\n",
+      "\n",
+      "|The| end|.|\n",
+      "--\n",
+      "Done agent: agent with output: Eugene thinks that cats like fish. Now let me tell you a story about that thought:\n",
+      "\n",
+      "Once upon a time, in a small village, there lived a curious cat named Whiskers. Whiskers was known for his love of fish. Every day, he would venture to the nearby river in search of his favorite meal.\n",
+      "\n",
+      "One sunny morning, Whiskers set out on his daily fish-hunting expedition. As he approached the river, he could smell the fresh scent of the water and feel the excitement building up inside him. Whiskers knew that today might be his lucky day.\n",
+      "\n",
+      "He carefully tiptoed along the riverbank, his eyes fixed on the water's surface. Suddenly, he spotted a shimmering fish swimming gracefully through the clear blue water. Whiskers crouched low, ready to pounce.\n",
+      "\n",
+      "With lightning speed, he leaped into the air, his paws outstretched towards the fish. Splash! Whiskers landed right in the middle of the river, causing ripples to spread in all directions. But he didn't care. All he wanted was that delicious fish.\n",
+      "\n",
+      "Whiskers chased the fish with all his might, darting through the water with elegance and precision. The fish swam gracefully, trying to escape Whiskers' determined pursuit. But the cat was relentless.\n",
+      "\n",
+      "After a few moments of intense chase, Whiskers finally managed to catch the fish in his paws. He triumphantly carried it to the riverbank, where he enjoyed his well-deserved feast. The taste of the fresh fish was heavenly, satisfying his hunger and bringing a contented smile to his face.\n",
+      "\n",
+      "From that day on, Whiskers became known as the legendary fish-catching cat in the village. People would often gather by the river to watch him in action, amazed by his agility and determination. Whiskers taught everyone the importance of perseverance and following one's passion, just like he pursued his love for fish.\n",
+      "\n",
+      "And so, Whiskers continued his fish-hunting adventures, spreading joy and inspiration to everyone he encountered. He proved that when you have a passion for something, nothing can stop you from achieving it – just like cats and their love for fish.\n",
+      "\n",
+      "The end.\n"
+     ]
+    }
+   ],
+   "source": [
+    "async for event in remote_runnable.astream_events(\n",
+    "    {\"input\": \"what does eugene think of cats? Then tell me a story about that thought.\"},\n",
+    "    version=\"v1\",\n",
+    "):\n",
+    "    kind = event[\"event\"]\n",
+    "    if kind == \"on_chain_start\":\n",
+    "        if (\n",
+    "            event[\"name\"] == \"agent\"\n",
+    "        ):  # Was assigned when creating the agent with `.with_config({\"run_name\": \"Agent\"})`\n",
+    "            print(\n",
+    "                f\"Starting agent: {event['name']} with input: {event['data'].get('input')}\"\n",
+    "            )\n",
+    "    elif kind == \"on_chain_end\":\n",
+    "        if (\n",
+    "            event[\"name\"] == \"agent\"\n",
+    "        ):  # Was assigned when creating the agent with `.with_config({\"run_name\": \"Agent\"})`\n",
+    "            print()\n",
+    "            print(\"--\")\n",
+    "            print(\n",
+    "                f\"Done agent: {event['name']} with output: {event['data'].get('output')['output']}\"\n",
+    "            )\n",
+    "    if kind == \"on_chat_model_stream\":\n",
+    "        content = event[\"data\"][\"chunk\"].content\n",
+    "        if content:\n",
+    "            # Empty content in the context of OpenAI means\n",
+    "            # that the model is asking for a tool to be invoked.\n",
+    "            # So we only print non-empty content\n",
+    "            print(content, end=\"|\")\n",
+    "    elif kind == \"on_tool_start\":\n",
+    "        print(\"--\")\n",
+    "        print(\n",
+    "            f\"Starting tool: {event['name']} with inputs: {event['data'].get('input')}\"\n",
+    "        )\n",
+    "    elif kind == \"on_tool_end\":\n",
+    "        print(f\"Done tool: {event['name']}\")\n",
+    "        print(f\"Tool output was: {event['data'].get('output')}\")\n",
+    "        print(\"--\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Stream log\n",
+    "\n",
+    "If you need acccess the individual llm tokens from an agent use `astream_log`. Please make sure that you set **streaming=True** on your LLM (see server code). For this to work, the LLM must also support streaming!"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "--\n",
+      "RunLogPatch({'op': 'replace',\n",
+      "  'path': '',\n",
+      "  'value': {'final_output': None,\n",
+      "            'id': '1213bf75-c5ff-401a-b3c5-b4663ec7dfca',\n",
+      "            'logs': {},\n",
+      "            'streamed_output': []}})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/RunnableSequence',\n",
+      "  'value': {'end_time': None,\n",
+      "            'final_output': None,\n",
+      "            'id': '7629fc73-a2fd-48fa-b9a2-c336e270180d',\n",
+      "            'metadata': {'__langserve_endpoint': 'stream_log',\n",
+      "                         '__langserve_version': '0.0.37',\n",
+      "                         '__useragent': 'python-httpx/0.25.2'},\n",
+      "            'name': 'RunnableSequence',\n",
+      "            'start_time': '2024-01-05T22:24:50.737+00:00',\n",
+      "            'streamed_output': [],\n",
+      "            'streamed_output_str': [],\n",
+      "            'tags': [],\n",
+      "            'type': 'chain'}})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/RunnableParallel<input,agent_scratchpad>',\n",
+      "  'value': {'end_time': None,\n",
+      "            'final_output': None,\n",
+      "            'id': '72d2e143-d977-4661-b901-430cae1ee739',\n",
+      "            'metadata': {'__langserve_endpoint': 'stream_log',\n",
+      "                         '__langserve_version': '0.0.37',\n",
+      "                         '__useragent': 'python-httpx/0.25.2'},\n",
+      "            'name': 'RunnableParallel<input,agent_scratchpad>',\n",
+      "            'start_time': '2024-01-05T22:24:50.738+00:00',\n",
+      "            'streamed_output': [],\n",
+      "            'streamed_output_str': [],\n",
+      "            'tags': ['seq:step:1'],\n",
+      "            'type': 'chain'}})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/<lambda>',\n",
+      "  'value': {'end_time': None,\n",
+      "            'final_output': None,\n",
+      "            'id': '7ec78a86-596f-4904-9888-45ef67302cb8',\n",
+      "            'metadata': {'__langserve_endpoint': 'stream_log',\n",
+      "                         '__langserve_version': '0.0.37',\n",
+      "                         '__useragent': 'python-httpx/0.25.2'},\n",
+      "            'name': '<lambda>',\n",
+      "            'start_time': '2024-01-05T22:24:50.739+00:00',\n",
+      "            'streamed_output': [],\n",
+      "            'streamed_output_str': [],\n",
+      "            'tags': ['map:key:input'],\n",
+      "            'type': 'chain'}})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/<lambda>:2',\n",
+      "  'value': {'end_time': None,\n",
+      "            'final_output': None,\n",
+      "            'id': '281ca194-3e0c-490c-b637-759f5d971769',\n",
+      "            'metadata': {'__langserve_endpoint': 'stream_log',\n",
+      "                         '__langserve_version': '0.0.37',\n",
+      "                         '__useragent': 'python-httpx/0.25.2'},\n",
+      "            'name': '<lambda>',\n",
+      "            'start_time': '2024-01-05T22:24:50.739+00:00',\n",
+      "            'streamed_output': [],\n",
+      "            'streamed_output_str': [],\n",
+      "            'tags': ['map:key:agent_scratchpad'],\n",
+      "            'type': 'chain'}})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/<lambda>/final_output',\n",
+      "  'value': {'output': 'what does eugene think of cats?'}},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/<lambda>/end_time',\n",
+      "  'value': '2024-01-05T22:24:50.740+00:00'})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/<lambda>:2/final_output',\n",
+      "  'value': {'output': []}},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/<lambda>:2/end_time',\n",
+      "  'value': '2024-01-05T22:24:50.740+00:00'})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/RunnableParallel<input,agent_scratchpad>/final_output',\n",
+      "  'value': {'agent_scratchpad': [],\n",
+      "            'input': 'what does eugene think of cats?'}},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/RunnableParallel<input,agent_scratchpad>/end_time',\n",
+      "  'value': '2024-01-05T22:24:50.740+00:00'})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatPromptTemplate',\n",
+      "  'value': {'end_time': None,\n",
+      "            'final_output': None,\n",
+      "            'id': '68c6ad9d-b9f0-47e0-9ea7-d0f170a22732',\n",
+      "            'metadata': {'__langserve_endpoint': 'stream_log',\n",
+      "                         '__langserve_version': '0.0.37',\n",
+      "                         '__useragent': 'python-httpx/0.25.2'},\n",
+      "            'name': 'ChatPromptTemplate',\n",
+      "            'start_time': '2024-01-05T22:24:50.741+00:00',\n",
+      "            'streamed_output': [],\n",
+      "            'streamed_output_str': [],\n",
+      "            'tags': ['seq:step:2'],\n",
+      "            'type': 'prompt'}})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatPromptTemplate/final_output',\n",
+      "  'value': {'messages': [SystemMessage(content='You are a helpful assistant.'),\n",
+      "                         HumanMessage(content='what does eugene think of cats?')]}},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/ChatPromptTemplate/end_time',\n",
+      "  'value': '2024-01-05T22:24:50.741+00:00'})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI',\n",
+      "  'value': {'end_time': None,\n",
+      "            'final_output': None,\n",
+      "            'id': '883e05a9-6b29-4494-92b3-51c72973cb54',\n",
+      "            'metadata': {'__langserve_endpoint': 'stream_log',\n",
+      "                         '__langserve_version': '0.0.37',\n",
+      "                         '__useragent': 'python-httpx/0.25.2'},\n",
+      "            'name': 'ChatOpenAI',\n",
+      "            'start_time': '2024-01-05T22:24:50.743+00:00',\n",
+      "            'streamed_output': [],\n",
+      "            'streamed_output_str': [],\n",
+      "            'tags': ['seq:step:3'],\n",
+      "            'type': 'llm'}})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/ChatOpenAI/streamed_output_str/-', 'value': ''},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='', additional_kwargs={'function_call': {'name': 'get_eugene_thoughts', 'arguments': ''}})})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/ChatOpenAI/streamed_output_str/-', 'value': ''},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='', additional_kwargs={'function_call': {'arguments': '{\\n'}})})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/ChatOpenAI/streamed_output_str/-', 'value': ''},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='', additional_kwargs={'function_call': {'arguments': ' '}})})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/ChatOpenAI/streamed_output_str/-', 'value': ''},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='', additional_kwargs={'function_call': {'arguments': ' \"'}})})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/ChatOpenAI/streamed_output_str/-', 'value': ''},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='', additional_kwargs={'function_call': {'arguments': 'query'}})})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/ChatOpenAI/streamed_output_str/-', 'value': ''},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='', additional_kwargs={'function_call': {'arguments': '\":'}})})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/ChatOpenAI/streamed_output_str/-', 'value': ''},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='', additional_kwargs={'function_call': {'arguments': ' \"'}})})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/ChatOpenAI/streamed_output_str/-', 'value': ''},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='', additional_kwargs={'function_call': {'arguments': 'cats'}})})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/ChatOpenAI/streamed_output_str/-', 'value': ''},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='', additional_kwargs={'function_call': {'arguments': '\"\\n'}})})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/ChatOpenAI/streamed_output_str/-', 'value': ''},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='', additional_kwargs={'function_call': {'arguments': '}'}})})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/ChatOpenAI/streamed_output_str/-', 'value': ''},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='')})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI/final_output',\n",
+      "  'value': LLMResult(generations=[[ChatGeneration(generation_info={'finish_reason': 'function_call'}, message=AIMessage(content='', additional_kwargs={'function_call': {'name': 'get_eugene_thoughts', 'arguments': '{\\n  \"query\": \"cats\"\\n}'}}))]], llm_output=None, run=None)},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI/end_time',\n",
+      "  'value': '2024-01-05T22:24:51.497+00:00'})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/OpenAIFunctionsAgentOutputParser',\n",
+      "  'value': {'end_time': None,\n",
+      "            'final_output': None,\n",
+      "            'id': 'f8052926-8eb1-4df1-a701-309ee48fd63c',\n",
+      "            'metadata': {'__langserve_endpoint': 'stream_log',\n",
+      "                         '__langserve_version': '0.0.37',\n",
+      "                         '__useragent': 'python-httpx/0.25.2'},\n",
+      "            'name': 'OpenAIFunctionsAgentOutputParser',\n",
+      "            'start_time': '2024-01-05T22:24:51.498+00:00',\n",
+      "            'streamed_output': [],\n",
+      "            'streamed_output_str': [],\n",
+      "            'tags': ['seq:step:4'],\n",
+      "            'type': 'parser'}})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/OpenAIFunctionsAgentOutputParser/final_output',\n",
+      "  'value': AgentActionMessageLog(tool='get_eugene_thoughts', tool_input={'query': 'cats'}, log=\"\\nInvoking: `get_eugene_thoughts` with `{'query': 'cats'}`\\n\\n\\n\", message_log=[AIMessage(content='', additional_kwargs={'function_call': {'name': 'get_eugene_thoughts', 'arguments': '{\\n  \"query\": \"cats\"\\n}'}})])},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/OpenAIFunctionsAgentOutputParser/end_time',\n",
+      "  'value': '2024-01-05T22:24:51.499+00:00'})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/RunnableSequence/final_output',\n",
+      "  'value': {'output': AgentActionMessageLog(tool='get_eugene_thoughts', tool_input={'query': 'cats'}, log=\"\\nInvoking: `get_eugene_thoughts` with `{'query': 'cats'}`\\n\\n\\n\", message_log=[AIMessage(content='', additional_kwargs={'function_call': {'name': 'get_eugene_thoughts', 'arguments': '{\\n  \"query\": \"cats\"\\n}'}})])}},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/RunnableSequence/end_time',\n",
+      "  'value': '2024-01-05T22:24:51.500+00:00'})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/streamed_output/-',\n",
+      "  'value': {'actions': [AgentActionMessageLog(tool='get_eugene_thoughts', tool_input={'query': 'cats'}, log=\"\\nInvoking: `get_eugene_thoughts` with `{'query': 'cats'}`\\n\\n\\n\", message_log=[AIMessage(content='', additional_kwargs={'function_call': {'name': 'get_eugene_thoughts', 'arguments': '{\\n  \"query\": \"cats\"\\n}'}})])],\n",
+      "            'messages': [AIMessage(content='', additional_kwargs={'function_call': {'name': 'get_eugene_thoughts', 'arguments': '{\\n  \"query\": \"cats\"\\n}'}})]}},\n",
+      " {'op': 'replace',\n",
+      "  'path': '/final_output',\n",
+      "  'value': {'actions': [AgentActionMessageLog(tool='get_eugene_thoughts', tool_input={'query': 'cats'}, log=\"\\nInvoking: `get_eugene_thoughts` with `{'query': 'cats'}`\\n\\n\\n\", message_log=[AIMessage(content='', additional_kwargs={'function_call': {'name': 'get_eugene_thoughts', 'arguments': '{\\n  \"query\": \"cats\"\\n}'}})])],\n",
+      "            'messages': [AIMessage(content='', additional_kwargs={'function_call': {'name': 'get_eugene_thoughts', 'arguments': '{\\n  \"query\": \"cats\"\\n}'}})]}})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/get_eugene_thoughts',\n",
+      "  'value': {'end_time': None,\n",
+      "            'final_output': None,\n",
+      "            'id': 'e745d7ec-5992-4ab8-a3ef-191342c6470c',\n",
+      "            'metadata': {'__langserve_endpoint': 'stream_log',\n",
+      "                         '__langserve_version': '0.0.37',\n",
+      "                         '__useragent': 'python-httpx/0.25.2'},\n",
+      "            'name': 'get_eugene_thoughts',\n",
+      "            'start_time': '2024-01-05T22:24:51.501+00:00',\n",
+      "            'streamed_output': [],\n",
+      "            'streamed_output_str': [],\n",
+      "            'tags': [],\n",
+      "            'type': 'tool'}})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/get_eugene_thoughts/final_output',\n",
+      "  'value': {'output': \"[Document(page_content='cats like fish'), \"\n",
+      "                      \"Document(page_content='dogs like sticks')]\"}},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/get_eugene_thoughts/end_time',\n",
+      "  'value': '2024-01-05T22:24:51.737+00:00'})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/streamed_output/-',\n",
+      "  'value': {'messages': [FunctionMessage(content=\"[Document(page_content='cats like fish'), Document(page_content='dogs like sticks')]\", name='get_eugene_thoughts')],\n",
+      "            'steps': [{'action': AgentActionMessageLog(tool='get_eugene_thoughts', tool_input={'query': 'cats'}, log=\"\\nInvoking: `get_eugene_thoughts` with `{'query': 'cats'}`\\n\\n\\n\", message_log=[AIMessage(content='', additional_kwargs={'function_call': {'name': 'get_eugene_thoughts', 'arguments': '{\\n  \"query\": \"cats\"\\n}'}})]),\n",
+      "                       'observation': [Document(page_content='cats like fish'),\n",
+      "                                       Document(page_content='dogs like sticks')]}]}},\n",
+      " {'op': 'add',\n",
+      "  'path': '/final_output/steps',\n",
+      "  'value': [{'action': AgentActionMessageLog(tool='get_eugene_thoughts', tool_input={'query': 'cats'}, log=\"\\nInvoking: `get_eugene_thoughts` with `{'query': 'cats'}`\\n\\n\\n\", message_log=[AIMessage(content='', additional_kwargs={'function_call': {'name': 'get_eugene_thoughts', 'arguments': '{\\n  \"query\": \"cats\"\\n}'}})]),\n",
+      "             'observation': [Document(page_content='cats like fish'),\n",
+      "                             Document(page_content='dogs like sticks')]}]},\n",
+      " {'op': 'add',\n",
+      "  'path': '/final_output/messages/1',\n",
+      "  'value': FunctionMessage(content=\"[Document(page_content='cats like fish'), Document(page_content='dogs like sticks')]\", name='get_eugene_thoughts')})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/RunnableSequence:2',\n",
+      "  'value': {'end_time': None,\n",
+      "            'final_output': None,\n",
+      "            'id': '15d4ae13-554b-4144-9aa8-05527a15c042',\n",
+      "            'metadata': {'__langserve_endpoint': 'stream_log',\n",
+      "                         '__langserve_version': '0.0.37',\n",
+      "                         '__useragent': 'python-httpx/0.25.2'},\n",
+      "            'name': 'RunnableSequence',\n",
+      "            'start_time': '2024-01-05T22:24:51.739+00:00',\n",
+      "            'streamed_output': [],\n",
+      "            'streamed_output_str': [],\n",
+      "            'tags': [],\n",
+      "            'type': 'chain'}})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/RunnableParallel<input,agent_scratchpad>:2',\n",
+      "  'value': {'end_time': None,\n",
+      "            'final_output': None,\n",
+      "            'id': '207dab98-d50c-4f2c-934d-291ba3ffd953',\n",
+      "            'metadata': {'__langserve_endpoint': 'stream_log',\n",
+      "                         '__langserve_version': '0.0.37',\n",
+      "                         '__useragent': 'python-httpx/0.25.2'},\n",
+      "            'name': 'RunnableParallel<input,agent_scratchpad>',\n",
+      "            'start_time': '2024-01-05T22:24:51.740+00:00',\n",
+      "            'streamed_output': [],\n",
+      "            'streamed_output_str': [],\n",
+      "            'tags': ['seq:step:1'],\n",
+      "            'type': 'chain'}})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/<lambda>:3',\n",
+      "  'value': {'end_time': None,\n",
+      "            'final_output': None,\n",
+      "            'id': '8795a642-3f8e-4e5c-aec6-825b7aac2148',\n",
+      "            'metadata': {'__langserve_endpoint': 'stream_log',\n",
+      "                         '__langserve_version': '0.0.37',\n",
+      "                         '__useragent': 'python-httpx/0.25.2'},\n",
+      "            'name': '<lambda>',\n",
+      "            'start_time': '2024-01-05T22:24:51.741+00:00',\n",
+      "            'streamed_output': [],\n",
+      "            'streamed_output_str': [],\n",
+      "            'tags': ['map:key:input'],\n",
+      "            'type': 'chain'}})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/<lambda>:4',\n",
+      "  'value': {'end_time': None,\n",
+      "            'final_output': None,\n",
+      "            'id': '60be3aac-3718-4386-a941-1dd371b9a3ea',\n",
+      "            'metadata': {'__langserve_endpoint': 'stream_log',\n",
+      "                         '__langserve_version': '0.0.37',\n",
+      "                         '__useragent': 'python-httpx/0.25.2'},\n",
+      "            'name': '<lambda>',\n",
+      "            'start_time': '2024-01-05T22:24:51.741+00:00',\n",
+      "            'streamed_output': [],\n",
+      "            'streamed_output_str': [],\n",
+      "            'tags': ['map:key:agent_scratchpad'],\n",
+      "            'type': 'chain'}})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/<lambda>:3/final_output',\n",
+      "  'value': {'output': 'what does eugene think of cats?'}},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/<lambda>:3/end_time',\n",
+      "  'value': '2024-01-05T22:24:51.742+00:00'})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/<lambda>:4/final_output',\n",
+      "  'value': {'output': [AIMessage(content='', additional_kwargs={'function_call': {'name': 'get_eugene_thoughts', 'arguments': '{\\n  \"query\": \"cats\"\\n}'}}),\n",
+      "                       FunctionMessage(content=\"[Document(page_content='cats like fish'), Document(page_content='dogs like sticks')]\", name='get_eugene_thoughts')]}},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/<lambda>:4/end_time',\n",
+      "  'value': '2024-01-05T22:24:51.742+00:00'})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/RunnableParallel<input,agent_scratchpad>:2/final_output',\n",
+      "  'value': {'agent_scratchpad': [AIMessage(content='', additional_kwargs={'function_call': {'name': 'get_eugene_thoughts', 'arguments': '{\\n  \"query\": \"cats\"\\n}'}}),\n",
+      "                                 FunctionMessage(content=\"[Document(page_content='cats like fish'), Document(page_content='dogs like sticks')]\", name='get_eugene_thoughts')],\n",
+      "            'input': 'what does eugene think of cats?'}},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/RunnableParallel<input,agent_scratchpad>:2/end_time',\n",
+      "  'value': '2024-01-05T22:24:51.742+00:00'})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatPromptTemplate:2',\n",
+      "  'value': {'end_time': None,\n",
+      "            'final_output': None,\n",
+      "            'id': 'f1ff89b0-ae9c-49d1-8a2e-73c7f001f1e7',\n",
+      "            'metadata': {'__langserve_endpoint': 'stream_log',\n",
+      "                         '__langserve_version': '0.0.37',\n",
+      "                         '__useragent': 'python-httpx/0.25.2'},\n",
+      "            'name': 'ChatPromptTemplate',\n",
+      "            'start_time': '2024-01-05T22:24:51.743+00:00',\n",
+      "            'streamed_output': [],\n",
+      "            'streamed_output_str': [],\n",
+      "            'tags': ['seq:step:2'],\n",
+      "            'type': 'prompt'}})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatPromptTemplate:2/final_output',\n",
+      "  'value': {'messages': [SystemMessage(content='You are a helpful assistant.'),\n",
+      "                         HumanMessage(content='what does eugene think of cats?'),\n",
+      "                         AIMessage(content='', additional_kwargs={'function_call': {'name': 'get_eugene_thoughts', 'arguments': '{\\n  \"query\": \"cats\"\\n}'}}),\n",
+      "                         FunctionMessage(content=\"[Document(page_content='cats like fish'), Document(page_content='dogs like sticks')]\", name='get_eugene_thoughts')]}},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/ChatPromptTemplate:2/end_time',\n",
+      "  'value': '2024-01-05T22:24:51.743+00:00'})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2',\n",
+      "  'value': {'end_time': None,\n",
+      "            'final_output': None,\n",
+      "            'id': 'c4aeb3d0-33c3-4f46-80e9-d1ffb0f983f2',\n",
+      "            'metadata': {'__langserve_endpoint': 'stream_log',\n",
+      "                         '__langserve_version': '0.0.37',\n",
+      "                         '__useragent': 'python-httpx/0.25.2'},\n",
+      "            'name': 'ChatOpenAI',\n",
+      "            'start_time': '2024-01-05T22:24:51.744+00:00',\n",
+      "            'streamed_output': [],\n",
+      "            'streamed_output_str': [],\n",
+      "            'tags': ['seq:step:3'],\n",
+      "            'type': 'llm'}})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/ChatOpenAI:2/streamed_output_str/-', 'value': ''},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='')})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output_str/-',\n",
+      "  'value': 'E'},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='E')})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output_str/-',\n",
+      "  'value': 'ug'},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='ug')})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output_str/-',\n",
+      "  'value': 'ene'},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='ene')})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output_str/-',\n",
+      "  'value': ' has'},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content=' has')})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output_str/-',\n",
+      "  'value': ' two'},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content=' two')})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output_str/-',\n",
+      "  'value': ' thoughts'},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content=' thoughts')})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output_str/-',\n",
+      "  'value': ' on'},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content=' on')})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output_str/-',\n",
+      "  'value': ' cats'},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content=' cats')})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output_str/-',\n",
+      "  'value': '.'},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='.')})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output_str/-',\n",
+      "  'value': ' One'},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content=' One')})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output_str/-',\n",
+      "  'value': ' thought'},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content=' thought')})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output_str/-',\n",
+      "  'value': ' is'},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content=' is')})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output_str/-',\n",
+      "  'value': ' \"'},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content=' \"')})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output_str/-',\n",
+      "  'value': 'cats'},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='cats')})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output_str/-',\n",
+      "  'value': ' like'},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content=' like')})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output_str/-',\n",
+      "  'value': ' fish'},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content=' fish')})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output_str/-',\n",
+      "  'value': '\"'},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='\"')})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output_str/-',\n",
+      "  'value': ' and'},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content=' and')})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output_str/-',\n",
+      "  'value': ' the'},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content=' the')})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output_str/-',\n",
+      "  'value': ' other'},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content=' other')})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output_str/-',\n",
+      "  'value': ' thought'},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content=' thought')})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output_str/-',\n",
+      "  'value': ' is'},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content=' is')})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output_str/-',\n",
+      "  'value': ' \"'},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content=' \"')})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output_str/-',\n",
+      "  'value': 'dogs'},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='dogs')})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output_str/-',\n",
+      "  'value': ' like'},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content=' like')})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output_str/-',\n",
+      "  'value': ' sticks'},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content=' sticks')})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output_str/-',\n",
+      "  'value': '\".'},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='\".')})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/ChatOpenAI:2/streamed_output_str/-', 'value': ''},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='')})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/final_output',\n",
+      "  'value': LLMResult(generations=[[ChatGeneration(text='Eugene has two thoughts on cats. One thought is \"cats like fish\" and the other thought is \"dogs like sticks\".', generation_info={'finish_reason': 'stop'}, message=AIMessage(content='Eugene has two thoughts on cats. One thought is \"cats like fish\" and the other thought is \"dogs like sticks\".'))]], llm_output=None, run=None)},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/end_time',\n",
+      "  'value': '2024-01-05T22:24:53.082+00:00'})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/OpenAIFunctionsAgentOutputParser:2',\n",
+      "  'value': {'end_time': None,\n",
+      "            'final_output': None,\n",
+      "            'id': '7074d8c2-df0b-4d18-9890-dd050c2f0ca9',\n",
+      "            'metadata': {'__langserve_endpoint': 'stream_log',\n",
+      "                         '__langserve_version': '0.0.37',\n",
+      "                         '__useragent': 'python-httpx/0.25.2'},\n",
+      "            'name': 'OpenAIFunctionsAgentOutputParser',\n",
+      "            'start_time': '2024-01-05T22:24:53.083+00:00',\n",
+      "            'streamed_output': [],\n",
+      "            'streamed_output_str': [],\n",
+      "            'tags': ['seq:step:4'],\n",
+      "            'type': 'parser'}})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/OpenAIFunctionsAgentOutputParser:2/final_output',\n",
+      "  'value': AgentFinish(return_values={'output': 'Eugene has two thoughts on cats. One thought is \"cats like fish\" and the other thought is \"dogs like sticks\".'}, log='Eugene has two thoughts on cats. One thought is \"cats like fish\" and the other thought is \"dogs like sticks\".')},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/OpenAIFunctionsAgentOutputParser:2/end_time',\n",
+      "  'value': '2024-01-05T22:24:53.084+00:00'})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/RunnableSequence:2/final_output',\n",
+      "  'value': {'output': AgentFinish(return_values={'output': 'Eugene has two thoughts on cats. One thought is \"cats like fish\" and the other thought is \"dogs like sticks\".'}, log='Eugene has two thoughts on cats. One thought is \"cats like fish\" and the other thought is \"dogs like sticks\".')}},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/RunnableSequence:2/end_time',\n",
+      "  'value': '2024-01-05T22:24:53.084+00:00'})\n",
+      "--\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/streamed_output/-',\n",
+      "  'value': {'messages': [AIMessage(content='Eugene has two thoughts on cats. One thought is \"cats like fish\" and the other thought is \"dogs like sticks\".')],\n",
+      "            'output': 'Eugene has two thoughts on cats. One thought is \"cats '\n",
+      "                      'like fish\" and the other thought is \"dogs like '\n",
+      "                      'sticks\".'}},\n",
+      " {'op': 'add',\n",
+      "  'path': '/final_output/output',\n",
+      "  'value': 'Eugene has two thoughts on cats. One thought is \"cats like fish\" '\n",
+      "           'and the other thought is \"dogs like sticks\".'},\n",
+      " {'op': 'add',\n",
+      "  'path': '/final_output/messages/2',\n",
+      "  'value': AIMessage(content='Eugene has two thoughts on cats. One thought is \"cats like fish\" and the other thought is \"dogs like sticks\".')})\n"
+     ]
+    }
+   ],
+   "source": [
+    "async for chunk in remote_runnable.astream_log({\"input\": \"what does eugene think of cats?\"}):\n",
+    "    print('--')\n",
+    "    print(chunk)"
+   ]
   }
  ],
  "metadata": {
@@ -132,7 +943,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.6"
+   "version": "3.10.8"
   }
  },
  "nbformat": 4,

examples/agent/server.py

@@ -1,5 +1,24 @@
 #!/usr/bin/env python
-"""Example LangChain server exposes a conversational retrieval chain."""
+"""Example LangChain server exposes a conversational retrieval agent.
+
+Relevant LangChain documentation:
+
+* Creating a custom agent: https://python.langchain.com/docs/modules/agents/how_to/custom_agent
+* Streaming with agents: https://python.langchain.com/docs/modules/agents/how_to/streaming#custom-streaming-with-events
+* General streaming documentation: https://python.langchain.com/docs/expression_language/streaming
+
+**ATTENTION**
+1. To support streaming individual tokens you will need to use the astream events
+   endpoint rather than the streaming endpoint.
+2. This example does not truncate message history, so it will crash if you
+   send too many messages (exceed token length).
+3. The playground at the moment does not render agent output well! If you want to
+   use the playground you need to customize it's output server side using astream
+   events by wrapping it within another runnable.
+4. See the client notebook it has an example of how to use stream_events client side!
+"""
+from typing import Any
+
 from fastapi import FastAPI
 from langchain.agents import AgentExecutor, tool
 from langchain.agents.format_scratchpad import format_to_openai_functions
@@ -30,12 +49,22 @@ tools = [get_eugene_thoughts]
 prompt = ChatPromptTemplate.from_messages(
     [
         ("system", "You are a helpful assistant."),
+        # Please note that the ordering of the user input vs.
+        # the agent_scratchpad is important.
+        # The agent_scratchpad is a working space for the agent to think,
+        # invoke tools, see tools outputs in order to respond to the given
+        # user input. It has to come AFTER the user input.
         ("user", "{input}"),
         MessagesPlaceholder(variable_name="agent_scratchpad"),
     ]
 )
 
-llm = ChatOpenAI()
+# We need to set streaming=True on the LLM to support streaming individual tokens.
+# Tokens will be available when using the stream_log / stream events endpoints,
+# but not when using the stream endpoint since the stream implementation for agent
+# streams action observation pairs not individual tokens.
+# See the client notebook that shows how to use the stream events endpoint.
+llm = ChatOpenAI(model="gpt-3.5-turbo", temperature=0, streaming=True)
 
 llm_with_tools = llm.bind(functions=[format_tool_to_openai_function(t) for t in tools])
 
@@ -56,7 +85,7 @@ agent_executor = AgentExecutor(agent=agent, tools=tools)
 app = FastAPI(
     title="LangChain Server",
     version="1.0",
-    description="Spin up a simple api server using Langchain's Runnable interfaces",
+    description="Spin up a simple api server using LangChain's Runnable interfaces",
 )
 
 
@@ -67,14 +96,20 @@ class Input(BaseModel):
 
 
 class Output(BaseModel):
-    output: str
+    output: Any
 
 
 # Adds routes to the app for using the chain under:
 # /invoke
 # /batch
 # /stream
-add_routes(app, agent_executor.with_types(input_type=Input, output_type=Output))
+# /stream_events
+add_routes(
+    app,
+    agent_executor.with_types(input_type=Input, output_type=Output).with_config(
+        {"run_name": "agent"}
+    ),
+)
 
 if __name__ == "__main__":
     import uvicorn

examples/agent_custom_streaming/client.ipynb

@@ -0,0 +1,232 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Client\n",
+    "\n",
+    "Client code interacting with a server that implements: \n",
+    "\n",
+    "* custom streaming for an agent\n",
+    "* agent with user selected tools\n",
+    "\n",
+    "This agent does not have memory! See other examples in LangServe to see how to add memory.\n",
+    "\n",
+    "**ATTENTION** We made the agent stream strings as an output. This is almost certainly not what you would want for your application. Feel free to adapt to return more structured output; however, keep in mind that likely the client can just use `astream_events`!\n",
+    "\n",
+    "See relevant documentation about agents:\n",
+    "\n",
+    "* Creating a custom agent: https://python.langchain.com/docs/modules/agents/how_to/custom_agent\n",
+    "* Streaming with agents: https://python.langchain.com/docs/modules/agents/how_to/streaming#custom-streaming-with-events\n",
+    "* General streaming documentation: https://python.langchain.com/docs/expression_language/streaming"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "You can interact with this via API directly"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "16"
+      ]
+     },
+     "execution_count": 1,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "word = \"audioeeeeeeeeeee\"\n",
+    "len(word)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "Starting agent: agent with input: {'input': 'what is the length of the word audioeeeeeeeeeee?'}\n",
+      "The length of the word \"audioeeeeeeeeeee\" is 15 characters.\n",
+      "Done agent: agent with output: The length of the word \"audioeeeeeeeeeee\" is 15 characters.\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "import requests\n",
+    "\n",
+    "inputs = {\"input\": {\"input\": f\"what is the length of the word {word}?\", \"chat_history\": [], \"tools\": []}}\n",
+    "response = requests.post(\"http://localhost:8000/invoke\", json=inputs)\n",
+    "\n",
+    "print(response.json()['output'])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Let's provide it with a tool to test that tool selection works"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "Starting agent: agent with input: {'input': 'what is the length of the word audioeeeeeeeeeee?'}\n",
+      "\n",
+      "Starting tool: word_length with inputs: {'word': 'audioeeeeeeeeeee'}\n",
+      "\n",
+      "Done tool: word_length with output: 16\n",
+      "The length of the word \"audioeeeeeeeeeee\" is 16.\n",
+      "Done agent: agent with output: The length of the word \"audioeeeeeeeeeee\" is 16.\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "import requests\n",
+    "\n",
+    "inputs = {\"input\": {\"input\": f\"what is the length of the word {word}?\", \"chat_history\": [], \"tools\": [\"word_length\", \"favorite_animal\"]}}\n",
+    "response = requests.post(\"http://localhost:8000/invoke\", json=inputs)\n",
+    "\n",
+    "print(response.json()['output'])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "You can also interact with this via the RemoteRunnable interface (to use in other chains)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "from langserve import RemoteRunnable\n",
+    "\n",
+    "remote_runnable = RemoteRunnable(\"http://localhost:8000/\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Remote runnable has the same interface as local runnables"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Stream\n",
+    "\n",
+    "Streaming output from a **CUSTOM STREAMING** implementation that streams string representations of intermediate steps. Please see server side implementation for details."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "|Starting agent: agent with input: {'input': 'What is eugenes favorite animal?'}|\n",
+      "|I|'m| sorry|,| but| I| don|'t| have| access| to| personal| information| about| individuals| unless| it| has| been| shared| with| me| in| the| course| of| our| conversation|.|\n",
+      "|Done agent: agent with output: I'm sorry, but I don't have access to personal information about individuals unless it has been shared with me in the course of our conversation.|\n",
+      "|"
+     ]
+    }
+   ],
+   "source": [
+    "async for chunk in remote_runnable.astream({\"input\": \"What is eugenes favorite animal?\", \"tools\": [\"word_length\"]}):\n",
+    "    print(chunk, end='|', flush=True)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "|Starting agent: agent with input: {'input': 'What is eugenes favorite animal?'}|\n",
+      "|\n",
+      "|Starting tool: favorite_animal with inputs: {'name': 'Eugene'}|\n",
+      "|\n",
+      "|Done tool: favorite_animal with output: cat|\n",
+      "|E|ug|ene|'s| favorite| animal| is| a| cat|.|\n",
+      "|Done agent: agent with output: Eugene's favorite animal is a cat.|\n",
+      "|"
+     ]
+    }
+   ],
+   "source": [
+    "async for chunk in remote_runnable.astream({\"input\": \"What is eugenes favorite animal?\", \"tools\": [\"word_length\", \"favorite_animal\"]}):\n",
+    "    print(chunk, end='|', flush=True)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.8"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

examples/agent_custom_streaming/server.py

@@ -0,0 +1,248 @@
+#!/usr/bin/env python
+"""Example LangChain server that shows how to customize streaming for an agent.
+
+Example uses a RunnableLambda that:
+
+1) Uses the agent's astream events method to create a custom streaming API endpoint.
+2) Instantiates an agent with custom tools (based on the user request).
+
+In this example, we kept things simple and are outputting strings to the client
+with all the intermediate steps of the agent. This is just for demonstration
+purposes, and usually you would want to return more structured output in the form
+of a dictionary.
+
+To add history to the agent you can use RunnableWithHistory. Please see the
+other examples in LangServe for how to use RunnableWithHistory to store history
+on the server side.
+
+Alternatively, you can keep track of history on the client side and send it to the
+server with each request. For that to work, you will definitely want to modify the
+streaming output to yield dictionaries with structured output, so it's easy
+to determine what the final agent output was on the client side.
+
+Customize the streaming output to your use case!
+
+Note that we configure the agent using the `tools` field in the input rather
+than using configurable fields. Using custom runnables and configurable fields
+is another option to customize the agent. 
+
+Please see configurable_agent_executor: https://github.com/langchain-ai/langserve/blob/main/examples/configurable_agent_executor/server.py
+for an example that uses a custom runnable with configurable fields.
+
+Relevant LangChain documentation:
+
+* Creating a custom agent: https://python.langchain.com/docs/modules/agents/how_to/custom_agent
+* Streaming with agents: https://python.langchain.com/docs/modules/agents/how_to/streaming#custom-streaming-with-events
+* General streaming documentation: https://python.langchain.com/docs/expression_language/streaming
+* Message History: https://python.langchain.com/docs/expression_language/how_to/message_history
+
+**ATTENTION**
+1. This example does not truncate message history, so it will crash if you
+   send too many messages (exceed token length).
+2. The playground at the moment does not render agent output well! If you want to
+   use the playground you need to customize it's output server side using astream
+   events by wrapping it within another runnable.
+3. See the client notebook to see how .stream() behaves!
+"""  # noqa: E501
+from typing import Any, AsyncIterator, List, Literal
+
+from fastapi import FastAPI
+from langchain.agents import AgentExecutor, tool
+from langchain.agents.format_scratchpad.openai_tools import (
+    format_to_openai_tool_messages,
+)
+from langchain.agents.output_parsers.openai_tools import OpenAIToolsAgentOutputParser
+from langchain.prompts import MessagesPlaceholder
+from langchain_community.tools.convert_to_openai import format_tool_to_openai_tool
+from langchain_core.prompts import ChatPromptTemplate
+from langchain_core.runnables import RunnableLambda
+from langchain_openai import ChatOpenAI
+
+from langserve import add_routes
+from langserve.pydantic_v1 import BaseModel
+
+prompt = ChatPromptTemplate.from_messages(
+    [
+        (
+            "system",
+            "You are very powerful assistant, but bad at calculating lengths of words. "
+            "Talk with the user as normal. "
+            "If they ask you to calculate the length of a word, use a tool",
+        ),
+        # Please note the ordering of the fields in the prompt!
+        # The correct ordering is:
+        # 1. user - the user's current input
+        # 2. agent_scratchpad - the agent's working space for thinking and
+        #    invoking tools to respond to the user's input.
+        # If you change the ordering, the agent will not work correctly since
+        # the messages will be shown to the underlying LLM in the wrong order.
+        ("user", "{input}"),
+        MessagesPlaceholder(variable_name="agent_scratchpad"),
+    ]
+)
+
+
+@tool
+def word_length(word: str) -> int:
+    """Returns a counter word"""
+    return len(word)
+
+
+@tool
+def favorite_animal(name: str) -> str:
+    """Get the favorite animal of the person with the given name"""
+    if name.lower().strip() == "eugene":
+        return "cat"
+    return "dog"
+
+
+# We need to set streaming=True on the LLM to support streaming individual tokens.
+# Tokens will be available when using the stream_log / stream events endpoints,
+# but not when using the stream endpoint since the stream implementation for agent
+# streams action observation pairs not individual tokens.
+# See the client notebook that shows how to use the stream events endpoint.
+llm = ChatOpenAI(model="gpt-3.5-turbo", temperature=0, streaming=True)
+
+TOOL_MAPPING = {
+    "word_length": word_length,
+    "favorite_animal": favorite_animal,
+}
+KnownTool = Literal["word_length", "favorite_animal"]
+
+
+def _create_agent_with_tools(requested_tools: List[KnownTool]) -> AgentExecutor:
+    """Create an agent with custom tools."""
+    tools = []
+
+    for requested_tool in requested_tools:
+        if requested_tool not in TOOL_MAPPING:
+            raise ValueError(f"Unknown tool: {requested_tool}")
+        tools.append(TOOL_MAPPING[requested_tool])
+
+    if tools:
+        llm_with_tools = llm.bind(
+            tools=[format_tool_to_openai_tool(tool) for tool in tools]
+        )
+    else:
+        llm_with_tools = llm
+
+    agent = (
+        {
+            "input": lambda x: x["input"],
+            "agent_scratchpad": lambda x: format_to_openai_tool_messages(
+                x["intermediate_steps"]
+            ),
+        }
+        | prompt
+        | llm_with_tools
+        | OpenAIToolsAgentOutputParser()
+    )
+    agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True).with_config(
+        {"run_name": "agent"}
+    )
+    return agent_executor
+
+
+app = FastAPI(
+    title="LangChain Server",
+    version="1.0",
+    description="Spin up a simple api server using LangChain's Runnable interfaces",
+)
+
+
+# We need to add these input/output schemas because the current AgentExecutor
+# is lacking in schemas.
+class Input(BaseModel):
+    input: str
+    tools: List[KnownTool]
+
+
+async def custom_stream(input: Input) -> AsyncIterator[str]:
+    """A custom runnable that can stream content.
+
+    Args:
+        input: The input to the agent. See the Input model for more details.
+
+    Yields:
+        strings that are streamed to the client.
+
+
+    Strings were chosen for simplicity, feel free to adapt to your use case.
+
+    You will almost certainly want to return more structured output in the form
+    of a dictionary, so it's easy to determine what the agent is doing without
+    parsing the output.
+
+    Before creating a custom streaming API, you should consider if you can use
+    the existing astream events API and customize the output on the client side
+    (potentially less overall work both server and client side).
+    """
+    agent_executor = _create_agent_with_tools(input["tools"])
+    async for event in agent_executor.astream_events(
+        {
+            "input": input["input"],
+        },
+        version="v1",
+    ):
+        kind = event["event"]
+        if kind == "on_chain_start":
+            if (
+                event["name"] == "agent"
+            ):  # matches `.with_config({"run_name": "Agent"})` in agent_executor
+                yield "\n"
+                yield (
+                    f"Starting agent: {event['name']} "
+                    f"with input: {event['data'].get('input')}"
+                )
+                yield "\n"
+        elif kind == "on_chain_end":
+            if (
+                event["name"] == "agent"
+            ):  # matches `.with_config({"run_name": "Agent"})` in agent_executor
+                yield "\n"
+                yield (
+                    f"Done agent: {event['name']} "
+                    f"with output: {event['data'].get('output')['output']}"
+                )
+                yield "\n"
+        if kind == "on_chat_model_stream":
+            content = event["data"]["chunk"].content
+            if content:
+                # Empty content in the context of OpenAI means
+                # that the model is asking for a tool to be invoked.
+                # So we only print non-empty content
+                yield content
+        elif kind == "on_tool_start":
+            yield "\n"
+            yield (
+                f"Starting tool: {event['name']} "
+                f"with inputs: {event['data'].get('input')}"
+            )
+            yield "\n"
+        elif kind == "on_tool_end":
+            yield "\n"
+            yield (
+                f"Done tool: {event['name']} "
+                f"with output: {event['data'].get('output')}"
+            )
+            yield "\n"
+
+
+class Output(BaseModel):
+    output: Any
+
+
+# Adds routes to the app for using the chain under:
+# /invoke
+# /batch
+# /stream
+# /stream_events
+add_routes(
+    app,
+    RunnableLambda(custom_stream),
+)
+
+if __name__ == "__main__":
+    import uvicorn
+
+    uvicorn.run(app, host="localhost", port=8000)

examples/agent_with_history/client.ipynb

@@ -0,0 +1,554 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Client\n",
+    "\n",
+    "Demo of a client interacting with a remote agent that can use history.\n",
+    "\n",
+    "See relevant documentation about agents:\n",
+    "\n",
+    "* Creating a custom agent: https://python.langchain.com/docs/modules/agents/how_to/custom_agent\n",
+    "* Streaming with agents: https://python.langchain.com/docs/modules/agents/how_to/streaming#custom-streaming-with-events\n",
+    "* General streaming documentation: https://python.langchain.com/docs/expression_language/streaming"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "You can interact with this via API directly"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'output': {'output': 'The length of the word \"audioee\" is 7.'},\n",
+       " 'callback_events': [],\n",
+       " 'metadata': {'run_id': '1847be77-f53c-40ba-b88d-06af3a598b6e'}}"
+      ]
+     },
+     "execution_count": 1,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "import requests\n",
+    "\n",
+    "inputs = {\"input\": {\"input\": \"what is the length of the word audioee?\", \"chat_history\": []}}\n",
+    "response = requests.post(\"http://localhost:8000/invoke\", json=inputs)\n",
+    "\n",
+    "response.json()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "You can also interact with this via the RemoteRunnable interface (to use in other chains)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "from langserve import RemoteRunnable\n",
+    "\n",
+    "remote_runnable = RemoteRunnable(\"http://localhost:8000/\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Remote runnable has the same interface as local runnables"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "from langchain_core.messages import HumanMessage, AIMessage"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdin",
+     "output_type": "stream",
+     "text": [
+      "Human (Q/q to quit):  hello\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "AI: Hello! How can I assist you today?\n"
+     ]
+    },
+    {
+     "name": "stdin",
+     "output_type": "stream",
+     "text": [
+      "Human (Q/q to quit):  my name is eugene\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "AI: Nice to meet you, Eugene! How can I help you today?\n"
+     ]
+    },
+    {
+     "name": "stdin",
+     "output_type": "stream",
+     "text": [
+      "Human (Q/q to quit):  what is my name\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "AI: Your name is Eugene.\n"
+     ]
+    },
+    {
+     "name": "stdin",
+     "output_type": "stream",
+     "text": [
+      "Human (Q/q to quit):  what is the length of my name\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "AI: The length of your name, Eugene, is 6 characters.\n"
+     ]
+    },
+    {
+     "name": "stdin",
+     "output_type": "stream",
+     "text": [
+      "Human (Q/q to quit):  q\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "AI: Bye bye human\n"
+     ]
+    }
+   ],
+   "source": [
+    "chat_history = []\n",
+    "\n",
+    "while True:\n",
+    "    human = input(\"Human (Q/q to quit): \")\n",
+    "    if human in {\"q\", \"Q\"}:\n",
+    "        print('AI: Bye bye human')\n",
+    "        break\n",
+    "    ai = await remote_runnable.ainvoke({\"input\": human, \"chat_history\": chat_history})\n",
+    "    print(f\"AI: {ai['output']}\")\n",
+    "    chat_history.extend([HumanMessage(content=human), AIMessage(content=ai['output'])])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Stream\n",
+    "\n",
+    "Please note that streaming alternates between actions and observations. It does not stream individual tokens!\n",
+    "\n",
+    "To stream individual tokens, we need to use the astream events endpoint (see below)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdin",
+     "output_type": "stream",
+     "text": [
+      "Human (Q/q to quit):  hello\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "AI: \n",
+      "Hello! How can I assist you today?\n",
+      "------\n"
+     ]
+    },
+    {
+     "name": "stdin",
+     "output_type": "stream",
+     "text": [
+      "Human (Q/q to quit):  my name is eugene\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "AI: \n",
+      "Nice to meet you, Eugene! How can I help you today?\n",
+      "------\n"
+     ]
+    },
+    {
+     "name": "stdin",
+     "output_type": "stream",
+     "text": [
+      "Human (Q/q to quit):  what is my name\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "AI: \n",
+      "Your name is Eugene.\n",
+      "------\n"
+     ]
+    },
+    {
+     "name": "stdin",
+     "output_type": "stream",
+     "text": [
+      "Human (Q/q to quit):  what's the length of my name?\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "AI: \n",
+      "Calling Tool ```word_length``` with input ```{'word': 'Eugene'}```\n",
+      "------\n",
+      "Got result: ```6```\n",
+      "------\n",
+      "The length of your name, Eugene, is 6 characters.\n",
+      "------\n"
+     ]
+    },
+    {
+     "name": "stdin",
+     "output_type": "stream",
+     "text": [
+      "Human (Q/q to quit):  q\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "AI: Bye bye human\n"
+     ]
+    }
+   ],
+   "source": [
+    "chat_history = []\n",
+    "\n",
+    "while True:\n",
+    "    human = input(\"Human (Q/q to quit): \")\n",
+    "    if human in {\"q\", \"Q\"}:\n",
+    "        print('AI: Bye bye human')\n",
+    "        break\n",
+    "\n",
+    "    ai = None\n",
+    "    print(\"AI: \")\n",
+    "    async for chunk in remote_runnable.astream({\"input\": human, \"chat_history\": chat_history}):\n",
+    "        # Agent Action\n",
+    "        if \"actions\" in chunk:\n",
+    "            for action in chunk[\"actions\"]:\n",
+    "                print(\n",
+    "                    f\"Calling Tool ```{action['tool']}``` with input ```{action['tool_input']}```\"\n",
+    "                )\n",
+    "        # Observation\n",
+    "        elif \"steps\" in chunk:\n",
+    "            for step in chunk[\"steps\"]:\n",
+    "                print(f\"Got result: ```{step['observation']}```\")\n",
+    "        # Final result\n",
+    "        elif \"output\" in chunk:\n",
+    "            print(chunk['output'])\n",
+    "            ai = AIMessage(content=chunk['output'])\n",
+    "        else:\n",
+    "            raise ValueError\n",
+    "        print(\"------\")        \n",
+    "    chat_history.extend([HumanMessage(content=human), ai])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Stream Events"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdin",
+     "output_type": "stream",
+     "text": [
+      "Human (Q/q to quit):  hello! my name is eugene\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "AI: \n",
+      "Starting agent: agent with input: {'input': 'hello! my name is eugene', 'chat_history': []}\n",
+      "Hello| Eugene|!| How| can| I| assist| you| today|?|\n",
+      "--\n",
+      "Done agent: agent with output: Hello Eugene! How can I assist you today?\n"
+     ]
+    },
+    {
+     "name": "stdin",
+     "output_type": "stream",
+     "text": [
+      "Human (Q/q to quit):  what's the length of my name?\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "AI: \n",
+      "Starting agent: agent with input: {'input': \"what's the length of my name?\", 'chat_history': []}\n",
+      "--\n",
+      "Starting tool: word_length with inputs: {'word': 'my name'}\n",
+      "Done tool: word_length\n",
+      "Tool output was: 7\n",
+      "--\n",
+      "The| length| of| your| name|,| \"|my| name|\",| is| |7| characters|.|\n",
+      "--\n",
+      "Done agent: agent with output: The length of your name, \"my name\", is 7 characters.\n"
+     ]
+    },
+    {
+     "name": "stdin",
+     "output_type": "stream",
+     "text": [
+      "Human (Q/q to quit):  could you tell me a long story about the length of my name?\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "AI: \n",
+      "Starting agent: agent with input: {'input': 'could you tell me a long story about the length of my name?', 'chat_history': []}\n",
+      "Once| upon| a| time|,| there| was| a| person| named| [|Your| Name|].| Now|,| [|Your| Name|]| had| a| very| unique| and| special| name|.| It| was| a| name| that| carried| a| lot| of| meaning| and| significance|.| People| often| wondered| about| the| length| of| [|Your| Name|]'|s| name| and| how| it| compared| to| others|.\n",
+      "\n",
+      "|One| day|,| [|Your| Name|]| decided| to| embark| on| a| journey| to| discover| the| true| length| of| their| name|.| They| traveled| far| and| wide|,| seeking| the| wisdom| of| s|ages| and| scholars| who| were| known| for| their| knowledge| of| names| and| their| lengths|.\n",
+      "\n",
+      "|Along| the| way|,| [|Your| Name|]| encountered| many| interesting| characters| who| had| their| own| stories| to| tell| about| the| lengths| of| their| names|.| Some| had| short| names| that| were| easy| to| remember|,| while| others| had| long| names| that| seemed| to| go| on| forever|.\n",
+      "\n",
+      "|As| [|Your| Name|]| continued| their| quest|,| they| came| across| a| wise| old| wizard| who| claimed| to| have| a| magical| tool| that| could| calculate| the| exact| length| of| any| name|.| Intr|ig|ued|,| [|Your| Name|]| approached| the| wizard| and| asked| for| their| assistance|.\n",
+      "\n",
+      "|The| wizard| took| out| a| mystical| device| and| asked| [|Your| Name|]| to| spell| out| their| name|.| [|Your| Name|]| eagerly| complied|,| carefully| en|unci|ating| each| letter|.| The| wizard| then| waved| the| device| over| the| name| and| muttered| an| inc|ant|ation|.\n",
+      "\n",
+      "|In| an| instant|,| the| device| displayed| the| length| of| [|Your| Name|]'|s| name|.| It| was| a| number| that| represented| the| total| count| of| characters| in| their| name|,| including| spaces| and| punctuation| marks|.| [|Your| Name|]| was| amazed| to| see| the| result| and| thanked| the| wizard| for| their| help|.\n",
+      "\n",
+      "|Ar|med| with| this| newfound| knowledge|,| [|Your| Name|]| returned| home| and| shared| their| story| with| friends| and| family|.| They| realized| that| the| length| of| their| name| was| not| just| a| random| number|,| but| a| reflection| of| their| identity| and| the| unique| journey| they| had| taken| to| discover| it|.\n",
+      "\n",
+      "|And| so|,| [|Your| Name|]| lived| happily| ever| after|,| cher|ishing| their| name| and| the| story| behind| its| length|.| They| understood| that| the| length| of| a| name| is| not| just| a| matter| of| counting| letters|,| but| a| testament| to| the| individual|ity| and| significance| of| each| person|'s| identity|.|\n",
+      "--\n",
+      "Done agent: agent with output: Once upon a time, there was a person named [Your Name]. Now, [Your Name] had a very unique and special name. It was a name that carried a lot of meaning and significance. People often wondered about the length of [Your Name]'s name and how it compared to others.\n",
+      "\n",
+      "One day, [Your Name] decided to embark on a journey to discover the true length of their name. They traveled far and wide, seeking the wisdom of sages and scholars who were known for their knowledge of names and their lengths.\n",
+      "\n",
+      "Along the way, [Your Name] encountered many interesting characters who had their own stories to tell about the lengths of their names. Some had short names that were easy to remember, while others had long names that seemed to go on forever.\n",
+      "\n",
+      "As [Your Name] continued their quest, they came across a wise old wizard who claimed to have a magical tool that could calculate the exact length of any name. Intrigued, [Your Name] approached the wizard and asked for their assistance.\n",
+      "\n",
+      "The wizard took out a mystical device and asked [Your Name] to spell out their name. [Your Name] eagerly complied, carefully enunciating each letter. The wizard then waved the device over the name and muttered an incantation.\n",
+      "\n",
+      "In an instant, the device displayed the length of [Your Name]'s name. It was a number that represented the total count of characters in their name, including spaces and punctuation marks. [Your Name] was amazed to see the result and thanked the wizard for their help.\n",
+      "\n",
+      "Armed with this newfound knowledge, [Your Name] returned home and shared their story with friends and family. They realized that the length of their name was not just a random number, but a reflection of their identity and the unique journey they had taken to discover it.\n",
+      "\n",
+      "And so, [Your Name] lived happily ever after, cherishing their name and the story behind its length. They understood that the length of a name is not just a matter of counting letters, but a testament to the individuality and significance of each person's identity.\n"
+     ]
+    },
+    {
+     "name": "stdin",
+     "output_type": "stream",
+     "text": [
+      "Human (Q/q to quit):  that's not what I wanted. My name is eugene. calculate the length of my name and tell me a story about the result.\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "AI: \n",
+      "Starting agent: agent with input: {'input': \"that's not what I wanted. My name is eugene. calculate the length of my name and tell me a story about the result.\", 'chat_history': []}\n",
+      "--\n",
+      "Starting tool: word_length with inputs: {'word': 'eugene'}\n",
+      "Done tool: word_length\n",
+      "Tool output was: 6\n",
+      "--\n",
+      "The| length| of| your| name|,| Eugene|,| is| |6| characters|.| Now|,| let| me| tell| you| a| story| about| the| result|.\n",
+      "\n",
+      "|Once| upon| a| time|,| in| a| land| far| away|,| there| was| a| young| boy| named| Eugene|.| He| had| a| special| power| -| the| power| to| bring| joy| and| laughter| to| everyone| he| met|.| Eugene|'s| name|,| with| its| |6| letters|,| perfectly| reflected| his| vibrant| and| energetic| personality|.\n",
+      "\n",
+      "|One| day|,| Eugene| decided| to| embark| on| a| grand| adventure|.| He| set| off| on| a| journey| to| spread| happiness| and| positivity| throughout| the| kingdom|.| Along| the| way|,| he| encountered| people| from| all| walks| of| life| -| from| humble| farmers| to| noble| knights|.\n",
+      "\n",
+      "|With| his| infectious| smile| and| kind| heart|,| Eugene| touched| the| lives| of| everyone| he| met|.| His| name|,| with| its| |6| letters|,| became| synonymous| with| love|,| compassion|,| and| joy|.| People| would| often| say|,| \"|If| you| want| to| experience| true| happiness|,| just| spend| a| moment| with| Eugene|.\"\n",
+      "\n",
+      "|As| Eugene| continued| his| journey|,| word| of| his| incredible| ability| to| bring| happiness| spread| far| and| wide|.| People| from| distant| lands| would| travel| for| miles| just| to| catch| a| glimpse| of| him|.| His| name|,| with| its| |6| letters|,| became| a| symbol| of| hope| and| inspiration|.\n",
+      "\n",
+      "|E|ug|ene|'s| story| serves| as| a| reminder| that| sometimes|,| the| simplest| things| can| have| the| greatest| impact|.| His| name|,| with| its| |6| letters|,| became| a| beacon| of| light| in| a| world| that| often| seemed| dark| and| glo|omy|.\n",
+      "\n",
+      "|And| so|,| Eugene|'s| adventure| continues|,| as| he| spreads| joy| and| happiness| wherever| he| goes|.| His| name|,| with| its| |6| letters|,| will| forever| be| et|ched| in| the| hearts| of| those| whose| lives| he| has| touched|.\n",
+      "\n",
+      "|The| end|.|\n",
+      "--\n",
+      "Done agent: agent with output: The length of your name, Eugene, is 6 characters. Now, let me tell you a story about the result.\n",
+      "\n",
+      "Once upon a time, in a land far away, there was a young boy named Eugene. He had a special power - the power to bring joy and laughter to everyone he met. Eugene's name, with its 6 letters, perfectly reflected his vibrant and energetic personality.\n",
+      "\n",
+      "One day, Eugene decided to embark on a grand adventure. He set off on a journey to spread happiness and positivity throughout the kingdom. Along the way, he encountered people from all walks of life - from humble farmers to noble knights.\n",
+      "\n",
+      "With his infectious smile and kind heart, Eugene touched the lives of everyone he met. His name, with its 6 letters, became synonymous with love, compassion, and joy. People would often say, \"If you want to experience true happiness, just spend a moment with Eugene.\"\n",
+      "\n",
+      "As Eugene continued his journey, word of his incredible ability to bring happiness spread far and wide. People from distant lands would travel for miles just to catch a glimpse of him. His name, with its 6 letters, became a symbol of hope and inspiration.\n",
+      "\n",
+      "Eugene's story serves as a reminder that sometimes, the simplest things can have the greatest impact. His name, with its 6 letters, became a beacon of light in a world that often seemed dark and gloomy.\n",
+      "\n",
+      "And so, Eugene's adventure continues, as he spreads joy and happiness wherever he goes. His name, with its 6 letters, will forever be etched in the hearts of those whose lives he has touched.\n",
+      "\n",
+      "The end.\n"
+     ]
+    },
+    {
+     "name": "stdin",
+     "output_type": "stream",
+     "text": [
+      "Human (Q/q to quit):  q\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "AI: Bye bye human\n"
+     ]
+    }
+   ],
+   "source": [
+    "chat_history = []\n",
+    "\n",
+    "while True:\n",
+    "    human = input(\"Human (Q/q to quit): \")\n",
+    "    if human in {\"q\", \"Q\"}:\n",
+    "        print('AI: Bye bye human')\n",
+    "        break\n",
+    "    ai = None\n",
+    "    print(\"AI: \")\n",
+    "    async for event in remote_runnable.astream_events(\n",
+    "        {\"input\": human, \"chat_history\": chat_history},\n",
+    "        version=\"v1\",\n",
+    "    ):\n",
+    "        kind = event[\"event\"]\n",
+    "        if kind == \"on_chain_start\":\n",
+    "            if (\n",
+    "                event[\"name\"] == \"agent\"\n",
+    "            ):  # Was assigned when creating the agent with `.with_config({\"run_name\": \"Agent\"})`\n",
+    "                print(\n",
+    "                    f\"Starting agent: {event['name']} with input: {event['data'].get('input')}\"\n",
+    "                )\n",
+    "        elif kind == \"on_chain_end\":\n",
+    "            if (\n",
+    "                event[\"name\"] == \"agent\"\n",
+    "            ):  # Was assigned when creating the agent with `.with_config({\"run_name\": \"Agent\"})`\n",
+    "                print()\n",
+    "                print(\"--\")\n",
+    "                print(\n",
+    "                    f\"Done agent: {event['name']} with output: {event['data'].get('output')['output']}\"\n",
+    "                )\n",
+    "        if kind == \"on_chat_model_stream\":\n",
+    "            content = event[\"data\"][\"chunk\"].content\n",
+    "            if content:\n",
+    "                # Empty content in the context of OpenAI means\n",
+    "                # that the model is asking for a tool to be invoked.\n",
+    "                # So we only print non-empty content\n",
+    "                print(content, end=\"|\")\n",
+    "        elif kind == \"on_tool_start\":\n",
+    "            print(\"--\")\n",
+    "            print(\n",
+    "                f\"Starting tool: {event['name']} with inputs: {event['data'].get('input')}\"\n",
+    "            )\n",
+    "        elif kind == \"on_tool_end\":\n",
+    "            print(f\"Done tool: {event['name']}\")\n",
+    "            print(f\"Tool output was: {event['data'].get('output')}\")\n",
+    "            print(\"--\")    \n"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.8"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

examples/agent_with_history/server.py

@@ -0,0 +1,157 @@
+#!/usr/bin/env python
+"""Example LangChain server exposes and agent that has conversation history.
+
+In this example, the history is stored entirely on the client's side.
+
+Please see other examples in LangServe on how to use RunnableWithHistory to
+store history on the server side.
+
+Relevant LangChain documentation:
+
+* Creating a custom agent: https://python.langchain.com/docs/modules/agents/how_to/custom_agent
+* Streaming with agents: https://python.langchain.com/docs/modules/agents/how_to/streaming#custom-streaming-with-events
+* General streaming documentation: https://python.langchain.com/docs/expression_language/streaming
+* Message History: https://python.langchain.com/docs/expression_language/how_to/message_history
+
+**ATTENTION**
+1. To support streaming individual tokens you will need to use the astream events
+   endpoint rather than the streaming endpoint.
+2. This example does not truncate message history, so it will crash if you
+   send too many messages (exceed token length).
+3. The playground at the moment does not render agent output well! If you want to
+   use the playground you need to customize it's output server side using astream
+   events by wrapping it within another runnable.
+4. See the client notebook it has an example of how to use stream_events client side!
+"""  # noqa: E501
+from typing import Any, List, Union
+
+from fastapi import FastAPI
+from langchain.agents import AgentExecutor, tool
+from langchain.agents.format_scratchpad.openai_tools import (
+    format_to_openai_tool_messages,
+)
+from langchain.agents.output_parsers.openai_tools import OpenAIToolsAgentOutputParser
+from langchain.prompts import MessagesPlaceholder
+from langchain_community.tools.convert_to_openai import format_tool_to_openai_tool
+from langchain_core.messages import AIMessage, FunctionMessage, HumanMessage
+from langchain_core.prompts import ChatPromptTemplate
+from langchain_openai import ChatOpenAI
+
+from langserve import add_routes
+from langserve.pydantic_v1 import BaseModel, Field
+
+prompt = ChatPromptTemplate.from_messages(
+    [
+        (
+            "system",
+            "You are very powerful assistant, but bad at calculating lengths of words. "
+            "Talk with the user as normal. "
+            "If they ask you to calculate the length of a word, use a tool",
+        ),
+        # Please note the ordering of the fields in the prompt!
+        # The correct ordering is:
+        # 1. history - the past messages between the user and the agent
+        # 2. user - the user's current input
+        # 3. agent_scratchpad - the agent's working space for thinking and
+        #    invoking tools to respond to the user's input.
+        # If you change the ordering, the agent will not work correctly since
+        # the messages will be shown to the underlying LLM in the wrong order.
+        MessagesPlaceholder(variable_name="chat_history"),
+        ("user", "{input}"),
+        MessagesPlaceholder(variable_name="agent_scratchpad"),
+    ]
+)
+
+
+@tool
+def word_length(word: str) -> int:
+    """Returns a counter word"""
+    return len(word)
+
+
+# We need to set streaming=True on the LLM to support streaming individual tokens.
+# Tokens will be available when using the stream_log / stream events endpoints,
+# but not when using the stream endpoint since the stream implementation for agent
+# streams action observation pairs not individual tokens.
+# See the client notebook that shows how to use the stream events endpoint.
+llm = ChatOpenAI(model="gpt-3.5-turbo", temperature=0, streaming=True)
+
+tools = [word_length]
+
+
+llm_with_tools = llm.bind(tools=[format_tool_to_openai_tool(tool) for tool in tools])
+
+# ATTENTION: For production use case, it's a good idea to trim the prompt to avoid
+#            exceeding the context window length used by the model.
+#
+# To fix that simply adjust the chain to trim the prompt in whatever way
+# is appropriate for your use case.
+# For example, you may want to keep the system message and the last 10 messages.
+# Or you may want to trim based on the number of tokens.
+# Or you may want to also summarize the messages to keep information about things
+# that were learned about the user.
+#
+# def prompt_trimmer(messages: List[Union[HumanMessage, AIMessage, FunctionMessage]]):
+#     '''Trims the prompt to a reasonable length.'''
+#     # Keep in mind that when trimming you may want to keep the system message!
+#     return messages[-10:] # Keep last 10 messages.
+
+agent = (
+    {
+        "input": lambda x: x["input"],
+        "agent_scratchpad": lambda x: format_to_openai_tool_messages(
+            x["intermediate_steps"]
+        ),
+        "chat_history": lambda x: x["chat_history"],
+    }
+    | prompt
+    # | prompt_trimmer # See comment above.
+    | llm_with_tools
+    | OpenAIToolsAgentOutputParser()
+)
+agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)
+
+
+app = FastAPI(
+    title="LangChain Server",
+    version="1.0",
+    description="Spin up a simple api server using LangChain's Runnable interfaces",
+)
+
+
+# We need to add these input/output schemas because the current AgentExecutor
+# is lacking in schemas.
+class Input(BaseModel):
+    input: str
+    # The field extra defines a chat widget.
+    # Please see documentation about widgets in the main README.
+    # The widget is used in the playground.
+    # Keep in mind that playground support for agents is not great at the moment.
+    # To get a better experience, you'll need to customize the streaming output
+    # for now.
+    chat_history: List[Union[HumanMessage, AIMessage, FunctionMessage]] = Field(
+        ...,
+        extra={"widget": {"type": "chat", "input": "input", "output": "output"}},
+    )
+
+
+class Output(BaseModel):
+    output: Any
+
+
+# Adds routes to the app for using the chain under:
+# /invoke
+# /batch
+# /stream
+# /stream_events
+add_routes(
+    app,
+    agent_executor.with_types(input_type=Input, output_type=Output).with_config(
+        {"run_name": "agent"}
+    ),
+)
+
+if __name__ == "__main__":
+    import uvicorn
+
+    uvicorn.run(app, host="localhost", port=8000)

examples/api_handler_examples/server.py

@@ -0,0 +1,125 @@
+"""An example that shows how to use the API handler directly.
+
+For this to work with RemoteClient, the routes must match those expected
+by the client; i.e., /invoke, /batch, /stream, etc. No trailing slashes should be used.
+"""
+from importlib import metadata
+from typing import Annotated
+
+from fastapi import Depends, FastAPI, Request, Response
+from langchain_core.runnables import RunnableLambda
+from sse_starlette import EventSourceResponse
+
+from langserve import APIHandler
+
+PYDANTIC_VERSION = metadata.version("pydantic")
+_PYDANTIC_MAJOR_VERSION: int = int(PYDANTIC_VERSION.split(".")[0])
+
+app = FastAPI(
+    title="LangChain Server",
+    version="1.0",
+    description="Spin up a simple api server using Langchain's Runnable interfaces",
+)
+
+
+##
+# Example 1 -- invoke, batch together with doc-generation
+# This endpoint shows how to expose `invoke` and `batch` using the APIHandler.
+# It also shows how to generate documentation properly so it works correctly
+# depending on Fast API and pydantic versions.
+def add_one(x: int) -> int:
+    """Add one to the given number."""
+    return x + 1
+
+
+chain = RunnableLambda(add_one)
+
+api_handler = APIHandler(chain, path="/simple")
+
+
+# First register the endpoints without documentation
+@app.post("/simple/invoke", include_in_schema=False)
+async def simple_invoke(request: Request) -> Response:
+    """Handle a request."""
+    # The API Handler validates the parts of the request
+    # that are used by the runnnable (e.g., input, config fields)
+    return await api_handler.invoke(request)
+
+
+@app.post("/simple/batch", include_in_schema=False)
+async def simple_batch(request: Request) -> Response:
+    """Handle a request."""
+    # The API Handler validates the parts of the request
+    # that are used by the runnnable (e.g., input, config fields)
+    return await api_handler.batch(request)
+
+
+# Here, we show how to populate the documentation for the endpoint.
+# Please note that this is done separately from the actual endpoint.
+# This happens due to two reasons:
+# 1. FastAPI does not support using pydantic.v1 models in the docs endpoint.
+#    "https://github.com/tiangolo/fastapi/issues/10360"
+#    LangChain uses pydantic.v1 models!
+# 2. Configurable Runnables have a *dynamic* schema, which means that
+#    the shape of the input depends on the config.
+#    In this case, the openapi schema is a best effort showing the documentation
+#    that will work for the default config (and any non-conflicting configs).
+if _PYDANTIC_MAJOR_VERSION == 1:  # Do not use in your own
+    # Add documentation
+    @app.post("/simple/invoke")
+    async def simple_invoke_docs(
+        request: api_handler.InvokeRequest,
+    ) -> api_handler.InvokeResponse:
+        """API endpoint used only for documentation purposes. Populate /docs endpoint"""
+        raise NotImplementedError(
+            "This endpoint is only used for documentation purposes"
+        )
+
+    @app.post("/simple/batch")
+    async def simple_batch_docs(
+        request: api_handler.BatchRequest,
+    ) -> api_handler.BatchResponse:
+        """API endpoint used only for documentation purposes. Populate /docs endpoint"""
+        raise NotImplementedError(
+            "This endpoint is only used for documentation purposes"
+        )
+
+else:
+    print(
+        "Skipping documentation generation for pydantic v2: "
+        "https://github.com/tiangolo/fastapi/issues/10360"
+    )
+
+
+##
+# Example 2 -- Expose `invoke` and `stream` using the API Handler.
+# Uses FastAPI Depends get a ready API handler.
+async def _get_api_handler() -> APIHandler:
+    """Prepare a RunnableLambda."""
+    return APIHandler(RunnableLambda(add_one), path="/v2")
+
+
+@app.post("/v2/invoke")
+async def v2_invoke(
+    request: Request, runnable: Annotated[APIHandler, Depends(_get_api_handler)]
+) -> Response:
+    """Handle invoke request."""
+    # The API Handler validates the parts of the request
+    # that are used by the runnnable (e.g., input, config fields)
+    return await runnable.invoke(request)
+
+
+@app.post("/v2/stream")
+async def v2_stream(
+    request: Request, runnable: Annotated[APIHandler, Depends(_get_api_handler)]
+) -> EventSourceResponse:
+    """Handle stream request."""
+    # The API Handler validates the parts of the request
+    # that are used by the runnnable (e.g., input, config fields)
+    return await runnable.stream(request)
+
+
+if __name__ == "__main__":
+    import uvicorn
+
+    uvicorn.run(app, host="localhost", port=8000)

examples/auth/api_handler/client.ipynb

@@ -0,0 +1,212 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Client\n",
+    "\n",
+    "This is an example client that interacts with the server that has \"auth\".\n",
+    "\n",
+    "Please reference appropriate documentation in the server code and in FastAPI to actually make this secure.\n",
+    "\n",
+    "\n",
+    "**ATTENTION** Only the invoke endpoint has been defined by the server! \n",
+    "So batch/stream won't work. If you want to add stream and batch, you can do so as well on the server side.\n",
+    "The server is implemented using the APIHandler, it's more flexible, but does require a bit more code. :)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Login as Alice"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "import requests\n",
+    "\n",
+    "response = requests.post(\"http://localhost:8000/token\", data={\"username\": \"alice\", \"password\": \"secret1\"})\n",
+    "result = response.json()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "token = result['access_token']"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "inputs = {\"input\": \"hello\"}\n",
+    "response = requests.post(\"http://localhost:8000/my_runnable/invoke\", \n",
+    "    json={\n",
+    "        'input': 'hello',\n",
+    "    },\n",
+    "    headers={\n",
+    "        'Authorization': f\"Bearer {token}\"\n",
+    "    }\n",
+    ")\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'output': [{'page_content': 'cats like mice',\n",
+       "   'metadata': {'owner_id': 'alice'},\n",
+       "   'type': 'Document'},\n",
+       "  {'page_content': 'cats like cheese',\n",
+       "   'metadata': {'owner_id': 'alice'},\n",
+       "   'type': 'Document'}],\n",
+       " 'callback_events': [],\n",
+       " 'metadata': {'run_id': '1732c9aa-c6d3-4736-b8ca-01265fa8ba06'}}"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "response.json()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "tags": []
+   },
+   "source": [
+    "You can also interact with this via the RemoteRunnable interface (to use in other chains)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "from langserve import RemoteRunnable\n",
+    "\n",
+    "remote_runnable = RemoteRunnable(\"http://localhost:8000/my_runnable\", headers={\"Authorization\": f\"Bearer {token}\"})"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[Document(page_content='cats like mice', metadata={'owner_id': 'alice'}),\n",
+       " Document(page_content='cats like cheese', metadata={'owner_id': 'alice'})]"
+      ]
+     },
+     "execution_count": 6,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "await remote_runnable.ainvoke(\"cat\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Login as John"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "import requests\n",
+    "\n",
+    "response = requests.post(\"http://localhost:8000/token\", data={\"username\": \"john\", \"password\": \"secret2\"})\n",
+    "token = response.json()['access_token']\n",
+    "remote_runnable = RemoteRunnable(\"http://localhost:8000/my_runnable\", headers={\"Authorization\": f\"Bearer {token}\"})"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[Document(page_content='i like walks by the ocean', metadata={'owner_id': 'john'}),\n",
+       " Document(page_content='dogs like sticks', metadata={'owner_id': 'john'}),\n",
+       " Document(page_content='my favorite food is cheese', metadata={'owner_id': 'john'})]"
+      ]
+     },
+     "execution_count": 8,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "await remote_runnable.ainvoke(\"water\")"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.6"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

examples/auth/api_handler/server.py

@@ -0,0 +1,285 @@
+#!/usr/bin/env python
+"""Example that shows how to use the underlying APIHandler class directly with Auth.
+
+This example shows how to apply logic based on the user's identity.
+
+You can build on these concepts to implement a more complex app:
+* Add endpoints that allow users to manage their documents.
+* Make a more complex runnable that does something with the retrieved documents; e.g.,
+  a conversational agent that responds to the user's input with the retrieved documents
+  (which are user specific documents).
+
+For authentication, we use a fake token that's the same as the username, adapting
+the following example from the FastAPI docs:
+
+https://fastapi.tiangolo.com/tutorial/security/simple-oauth2/
+
+**ATTENTION**
+
+This example is not actually secure and should not be used in production.
+
+Once you understand how to use `per_req_config_modifier`, read through
+the FastAPI docs and implement proper auth:
+https://fastapi.tiangolo.com/tutorial/security/oauth2-jwt/
+
+
+**ATTENTION**
+
+This example does not integrate auth with OpenAPI, so the OpenAPI docs won't
+be able to help with authentication. This is currently a limitation
+if using `add_routes`. If you need this functionality, you can use
+the underlying `APIHandler` class directly, which affords maximal flexibility.
+"""
+from importlib import metadata
+from typing import Any, List, Optional, Union
+
+from fastapi import Depends, FastAPI, HTTPException, Request, Response, status
+from fastapi.security import OAuth2PasswordBearer, OAuth2PasswordRequestForm
+from langchain_community.embeddings.openai import OpenAIEmbeddings
+from langchain_community.vectorstores.chroma import Chroma
+from langchain_core.documents import Document
+from langchain_core.runnables import (
+    ConfigurableField,
+    RunnableConfig,
+    RunnableSerializable,
+)
+from langchain_core.vectorstores import VectorStore
+from typing_extensions import Annotated
+
+from langserve import APIHandler
+from langserve.pydantic_v1 import BaseModel
+
+
+class User(BaseModel):
+    username: str
+    email: Union[str, None] = None
+    full_name: Union[str, None] = None
+    disabled: Union[bool, None] = None
+
+
+class UserInDB(User):
+    hashed_password: str
+
+
+oauth2_scheme = OAuth2PasswordBearer(tokenUrl="token")
+
+app = FastAPI()
+
+FAKE_USERS_DB = {
+    "alice": {
+        "username": "alice",
+        "full_name": "Alice Wonderson",
+        "email": "alice@example.com",
+        "hashed_password": "fakehashedsecret1",
+        "disabled": False,
+    },
+    "john": {
+        "username": "john",
+        "full_name": "John Doe",
+        "email": "johndoe@example.com",
+        "hashed_password": "fakehashedsecret2",
+        "disabled": False,
+    },
+    "bob": {
+        "username": "john",
+        "full_name": "John Doe",
+        "email": "johndoe@example.com",
+        "hashed_password": "fakehashedsecret3",
+        "disabled": True,
+    },
+}
+
+
+def _fake_hash_password(password: str) -> str:
+    """Fake a hashed password."""
+    return "fakehashed" + password
+
+
+def _get_user(db: dict, username: str) -> Union[UserInDB, None]:
+    if username in db:
+        user_dict = db[username]
+        return UserInDB(**user_dict)
+
+
+def _fake_decode_token(token: str) -> Union[User, None]:
+    # This doesn't provide any security at all
+    # Check the next version
+    user = _get_user(FAKE_USERS_DB, token)
+    return user
+
+
+@app.post("/token")
+async def login(form_data: Annotated[OAuth2PasswordRequestForm, Depends()]):
+    user_dict = FAKE_USERS_DB.get(form_data.username)
+    if not user_dict:
+        raise HTTPException(status_code=400, detail="Incorrect username or password")
+    user = UserInDB(**user_dict)
+    hashed_password = _fake_hash_password(form_data.password)
+    if not hashed_password == user.hashed_password:
+        raise HTTPException(status_code=400, detail="Incorrect username or password")
+
+    return {"access_token": user.username, "token_type": "bearer"}
+
+
+async def get_current_user(token: Annotated[str, Depends(oauth2_scheme)]):
+    user = _fake_decode_token(token)
+    if not user:
+        raise HTTPException(
+            status_code=status.HTTP_401_UNAUTHORIZED,
+            detail="Invalid authentication credentials",
+            headers={"WWW-Authenticate": "Bearer"},
+        )
+    return user
+
+
+async def get_current_active_user(
+    current_user: Annotated[User, Depends(get_current_user)],
+):
+    if current_user.disabled:
+        raise HTTPException(status_code=400, detail="Inactive user")
+    return current_user
+
+
+class PerUserVectorstore(RunnableSerializable):
+    """A custom runnable that returns a list of documents for the given user.
+
+    The runnable is configurable by the user, and the search results are
+    filtered by the user ID.
+    """
+
+    user_id: Optional[str]
+    vectorstore: VectorStore
+
+    class Config:
+        # Allow arbitrary types since VectorStore is an abstract interface
+        # and not a pydantic model
+        arbitrary_types_allowed = True
+
+    def _invoke(
+        self, input: str, config: Optional[RunnableConfig] = None, **kwargs: Any
+    ) -> List[Document]:
+        """Invoke the retriever."""
+        # WARNING: Verify documentation of underlying vectorstore to make
+        # sure that it actually uses filters.
+        # Highly recommended to use unit-tests to verify this behavior, as
+        # implementations can be different depending on the underlying vectorstore.
+        retriever = self.vectorstore.as_retriever(
+            search_kwargs={"filter": {"owner_id": self.user_id}}
+        )
+        return retriever.invoke(input, config=config)
+
+    def invoke(
+        self, input: str, config: Optional[RunnableConfig] = None, **kwargs
+    ) -> List[Document]:
+        """Add one to an integer."""
+        return self._call_with_config(self._invoke, input, config, **kwargs)
+
+
+vectorstore = Chroma(
+    collection_name="some_collection",
+    embedding_function=OpenAIEmbeddings(),
+)
+
+vectorstore.add_documents(
+    [
+        Document(
+            page_content="cats like cheese",
+            metadata={"owner_id": "alice"},
+        ),
+        Document(
+            page_content="cats like mice",
+            metadata={"owner_id": "alice"},
+        ),
+        Document(
+            page_content="dogs like sticks",
+            metadata={"owner_id": "john"},
+        ),
+        Document(
+            page_content="my favorite food is cheese",
+            metadata={"owner_id": "john"},
+        ),
+        Document(
+            page_content="i like walks by the ocean",
+            metadata={"owner_id": "john"},
+        ),
+        Document(
+            page_content="dogs like grass",
+            metadata={"owner_id": "bob"},
+        ),
+    ]
+)
+
+per_user_retriever = PerUserVectorstore(
+    user_id=None,  # Placeholder ID that will be replaced by the per_req_config_modifier
+    vectorstore=vectorstore,
+).configurable_fields(
+    # Attention: Make sure to override the user ID for each request in the
+    # per_req_config_modifier. This should not be client configurable.
+    user_id=ConfigurableField(
+        id="user_id",
+        name="User ID",
+        description="The user ID to use for the retriever.",
+    )
+)
+
+
+# Let's define the API Handler
+api_handler = APIHandler(
+    per_user_retriever,
+    # Namespace for the runnable.
+    # Endpoints like batch / invoke should be under /my_runnable/invoke
+    # and /my_runnable/batch etc.
+    path="/my_runnable",
+)
+
+
+PYDANTIC_VERSION = metadata.version("pydantic")
+_PYDANTIC_MAJOR_VERSION: int = int(PYDANTIC_VERSION.split(".")[0])
+
+
+# **ATTENTION** Your code does not need to include both versions.
+# Use whichever version is appropriate given the pydantic version you are using.
+# Both versions are included here for demonstration purposes.
+#
+# If using pydantic <2, everything works as expected.
+# However, when using pydantic >=2 is installed, things are a bit
+# more complicated because LangChain uses the pydantic.v1 namespace
+# But the pydantic.v1 namespace is not supported by FastAPI.
+# See this issue: https://github.com/tiangolo/fastapi/issues/10360
+# So when using pydantic >=2, we need to use a vanilla starlette request
+# and response, and we will not have documentation.
+# Or we can create custom models for the request and response.
+# The underlying API Handler will still validate the request
+# correctly even if vanilla requests are used.
+if _PYDANTIC_MAJOR_VERSION == 1:
+
+    @app.post("/my_runnable/invoke")
+    async def invoke_with_auth(
+        # Included for documentation purposes
+        invoke_request: api_handler.InvokeRequest,
+        request: Request,
+        current_user: Annotated[User, Depends(get_current_active_user)],
+    ) -> Response:
+        """Handle a request."""
+        # The API Handler validates the parts of the request
+        # that are used by the runnnable (e.g., input, config fields)
+        config = {"configurable": {"user_id": current_user.username}}
+        return await api_handler.invoke(request, server_config=config)
+else:
+
+    @app.post("/my_runnable/invoke")
+    async def invoke_with_auth(
+        request: Request,
+        current_user: Annotated[User, Depends(get_current_active_user)],
+    ) -> Response:
+        """Handle a request."""
+        # The API Handler validates the parts of the request
+        # that are used by the runnnable (e.g., input, config fields)
+        config = {"configurable": {"user_id": current_user.username}}
+        return await api_handler.invoke(request, server_config=config)
+
+
+if __name__ == "__main__":
+    import uvicorn
+
+    uvicorn.run(app, host="localhost", port=8000)

examples/auth/global_deps/server.py

@@ -0,0 +1,53 @@
+#!/usr/bin/env python
+"""An example that uses Fast API global dependencies.
+
+This approach can be used if the same authentication logic can be used
+for all endpoints in the application.
+
+This may be a reasonable approach for simple applications.
+
+See:
+
+* https://fastapi.tiangolo.com/tutorial/dependencies/global-dependencies/
+* https://fastapi.tiangolo.com/tutorial/dependencies/
+* https://fastapi.tiangolo.com/tutorial/security/
+"""
+
+from fastapi import Depends, FastAPI, Header, HTTPException
+from langchain_core.runnables import RunnableLambda
+from typing_extensions import Annotated
+
+from langserve import add_routes
+
+
+async def verify_token(x_token: Annotated[str, Header()]) -> None:
+    """Verify the token is valid."""
+    # Replace this with your actual authentication logic
+    if x_token != "secret-token":
+        raise HTTPException(status_code=400, detail="X-Token header invalid")
+
+
+app = FastAPI(
+    title="LangChain Server",
+    version="1.0",
+    dependencies=[Depends(verify_token)],
+)
+
+
+def add_one(x: int) -> int:
+    """Add one to an integer."""
+    return x + 1
+
+
+chain = RunnableLambda(add_one)
+
+
+add_routes(
+    app,
+    chain,
+)
+
+if __name__ == "__main__":
+    import uvicorn
+
+    uvicorn.run(app, host="localhost", port=8000)

examples/auth/path_dependencies/server.py

@@ -0,0 +1,50 @@
+#!/usr/bin/env python
+"""An example that shows how to use path dependencies for authentication.
+
+The path dependencies are applied to all the routes added by the `add_routes`.
+
+To keep this example brief, we're providing a placeholder verify_token function
+that shows how to use path dependencies.
+
+To implement proper auth, please see the FastAPI docs:
+
+* https://fastapi.tiangolo.com/tutorial/dependencies/dependencies-in-path-operation-decorators/
+* https://fastapi.tiangolo.com/tutorial/dependencies/
+* https://fastapi.tiangolo.com/tutorial/security/
+"""  # noqa: E501
+
+from fastapi import Depends, FastAPI, Header, HTTPException
+from langchain_core.runnables import RunnableLambda
+from typing_extensions import Annotated
+
+from langserve import add_routes
+
+
+async def verify_token(x_token: Annotated[str, Header()]) -> None:
+    """Verify the token is valid."""
+    # Replace this with your actual authentication logic
+    if x_token != "secret-token":
+        raise HTTPException(status_code=400, detail="X-Token header invalid")
+
+
+app = FastAPI()
+
+
+def add_one(x: int) -> int:
+    """Add one to an integer."""
+    return x + 1
+
+
+chain = RunnableLambda(add_one)
+
+
+add_routes(
+    app,
+    chain,
+    dependencies=[Depends(verify_token)],
+)
+
+if __name__ == "__main__":
+    import uvicorn
+
+    uvicorn.run(app, host="localhost", port=8000)

examples/auth/per_req_config_modifier/client.ipynb

@@ -0,0 +1,207 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Client\n",
+    "\n",
+    "This is an example client that interacts with the server that has \"auth\".\n",
+    "\n",
+    "Please reference appropriate documentation in the server code and in FastAPI to actually make this secure."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Login as Alice"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "import requests\n",
+    "\n",
+    "response = requests.post(\"http://localhost:8000/token\", data={\"username\": \"alice\", \"password\": \"secret1\"})\n",
+    "result = response.json()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "token = result['access_token']"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "inputs = {\"input\": \"hello\"}\n",
+    "response = requests.post(\"http://localhost:8000/invoke\", \n",
+    "    json={\n",
+    "        'input': 'hello',\n",
+    "    },\n",
+    "    headers={\n",
+    "        'Authorization': f\"Bearer {token}\"\n",
+    "    }\n",
+    ")\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'output': [{'page_content': 'cats like mice',\n",
+       "   'metadata': {'owner_id': 'alice'},\n",
+       "   'type': 'Document'},\n",
+       "  {'page_content': 'cats like cheese',\n",
+       "   'metadata': {'owner_id': 'alice'},\n",
+       "   'type': 'Document'}],\n",
+       " 'callback_events': [],\n",
+       " 'metadata': {'run_id': '00000000-0000-0000-0000-000000000000'}}"
+      ]
+     },
+     "execution_count": 9,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "response.json()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "tags": []
+   },
+   "source": [
+    "You can also interact with this via the RemoteRunnable interface (to use in other chains)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "from langserve import RemoteRunnable\n",
+    "\n",
+    "remote_runnable = RemoteRunnable(\"http://localhost:8000/\", headers={\"Authorization\": f\"Bearer {token}\"})"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[Document(page_content='cats like mice', metadata={'owner_id': 'alice'}),\n",
+       " Document(page_content='cats like cheese', metadata={'owner_id': 'alice'})]"
+      ]
+     },
+     "execution_count": 11,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "await remote_runnable.ainvoke(\"cat\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Login as John"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 23,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "import requests\n",
+    "\n",
+    "response = requests.post(\"http://localhost:8000/token\", data={\"username\": \"john\", \"password\": \"secret2\"})\n",
+    "token = response.json()['access_token']\n",
+    "remote_runnable = RemoteRunnable(\"http://localhost:8000/\", headers={\"Authorization\": f\"Bearer {token}\"})"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 25,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[Document(page_content='i like walks by the ocean', metadata={'owner_id': 'john'}),\n",
+       " Document(page_content='dogs like sticks', metadata={'owner_id': 'john'}),\n",
+       " Document(page_content='my favorite food is cheese', metadata={'owner_id': 'john'})]"
+      ]
+     },
+     "execution_count": 25,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "await remote_runnable.ainvoke(\"water\")"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.6"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

examples/auth/per_req_config_modifier/server.py

@@ -0,0 +1,243 @@
+#!/usr/bin/env python
+"""Example that shows how to use `per_req_config_modifier`.
+
+This is a simple example that shows how to use configurable runnables with
+per request configuration modification to achieve behavior that's different
+depending on the user.
+
+You can build on these concepts to implement a more complex app:
+* Add endpoints that allow users to manage their documents.
+* Make a more complex runnable that does something with the retrieved documents; e.g.,
+  a conversational agent that responds to the user's input with the retrieved documents
+  (which are user specific documents).
+
+For authentication, we use a fake token that's the same as the username, adapting
+the following example from the FastAPI docs:
+
+https://fastapi.tiangolo.com/tutorial/security/simple-oauth2/
+
+**ATTENTION**
+
+This example is not actually secure and should not be used in production.
+
+Once you understand how to use `per_req_config_modifier`, read through
+the FastAPI docs and implement proper auth:
+https://fastapi.tiangolo.com/tutorial/security/oauth2-jwt/
+
+
+**ATTENTION**
+
+This example does not integrate auth with OpenAPI, so the OpenAPI docs won't
+be able to help with authentication. This is currently a limitation
+if using `add_routes`. If you need this functionality, you can use
+the underlying `APIHandler` class directly, which affords maximal flexibility.
+"""
+from typing import Any, Dict, List, Optional, Union
+
+from fastapi import Depends, FastAPI, HTTPException, Request, status
+from fastapi.security import OAuth2PasswordBearer, OAuth2PasswordRequestForm
+from langchain_community.embeddings.openai import OpenAIEmbeddings
+from langchain_community.vectorstores.chroma import Chroma
+from langchain_core.documents import Document
+from langchain_core.runnables import (
+    ConfigurableField,
+    RunnableConfig,
+    RunnableSerializable,
+)
+from langchain_core.vectorstores import VectorStore
+from typing_extensions import Annotated
+
+from langserve import add_routes
+from langserve.pydantic_v1 import BaseModel
+
+
+class User(BaseModel):
+    username: str
+    email: Union[str, None] = None
+    full_name: Union[str, None] = None
+    disabled: Union[bool, None] = None
+
+
+class UserInDB(User):
+    hashed_password: str
+
+
+oauth2_scheme = OAuth2PasswordBearer(tokenUrl="token")
+
+app = FastAPI()
+
+FAKE_USERS_DB = {
+    "alice": {
+        "username": "alice",
+        "full_name": "Alice Wonderson",
+        "email": "alice@example.com",
+        "hashed_password": "fakehashedsecret1",
+        "disabled": False,
+    },
+    "john": {
+        "username": "john",
+        "full_name": "John Doe",
+        "email": "johndoe@example.com",
+        "hashed_password": "fakehashedsecret2",
+        "disabled": False,
+    },
+    "bob": {
+        "username": "john",
+        "full_name": "John Doe",
+        "email": "johndoe@example.com",
+        "hashed_password": "fakehashedsecret3",
+        "disabled": True,
+    },
+}
+
+
+def _fake_hash_password(password: str) -> str:
+    """Fake a hashed password."""
+    return "fakehashed" + password
+
+
+def _get_user(db: dict, username: str) -> Union[UserInDB, None]:
+    if username in db:
+        user_dict = db[username]
+        return UserInDB(**user_dict)
+
+
+def _fake_decode_token(token: str) -> Union[User, None]:
+    # This doesn't provide any security at all
+    # Check the next version
+    user = _get_user(FAKE_USERS_DB, token)
+    return user
+
+
+@app.post("/token")
+async def login(form_data: Annotated[OAuth2PasswordRequestForm, Depends()]):
+    user_dict = FAKE_USERS_DB.get(form_data.username)
+    if not user_dict:
+        raise HTTPException(status_code=400, detail="Incorrect username or password")
+    user = UserInDB(**user_dict)
+    hashed_password = _fake_hash_password(form_data.password)
+    if not hashed_password == user.hashed_password:
+        raise HTTPException(status_code=400, detail="Incorrect username or password")
+
+    return {"access_token": user.username, "token_type": "bearer"}
+
+
+async def get_current_active_user_from_request(request: Request) -> User:
+    """Get the current active user from the request."""
+    token = await oauth2_scheme(request)
+    user = _fake_decode_token(token)
+    if not user:
+        raise HTTPException(
+            status_code=status.HTTP_401_UNAUTHORIZED,
+            detail="Invalid authentication credentials",
+            headers={"WWW-Authenticate": "Bearer"},
+        )
+    if user.disabled:
+        raise HTTPException(status_code=400, detail="Inactive user")
+    return user
+
+
+class PerUserVectorstore(RunnableSerializable):
+    """A custom runnable that returns a list of documents for the given user.
+
+    The runnable is configurable by the user, and the search results are
+    filtered by the user ID.
+    """
+
+    user_id: Optional[str]
+    vectorstore: VectorStore
+
+    class Config:
+        # Allow arbitrary types since VectorStore is an abstract interface
+        # and not a pydantic model
+        arbitrary_types_allowed = True
+
+    def _invoke(
+        self, input: str, config: Optional[RunnableConfig] = None, **kwargs: Any
+    ) -> List[Document]:
+        """Invoke the retriever."""
+        # WARNING: Verify documentation of underlying vectorstore to make
+        # sure that it actually uses filters.
+        # Highly recommended to use unit-tests to verify this behavior, as
+        # implementations can be different depending on the underlying vectorstore.
+        retriever = self.vectorstore.as_retriever(
+            search_kwargs={"filter": {"owner_id": self.user_id}}
+        )
+        return retriever.invoke(input, config=config)
+
+    def invoke(
+        self, input: str, config: Optional[RunnableConfig] = None, **kwargs
+    ) -> List[Document]:
+        """Add one to an integer."""
+        return self._call_with_config(self._invoke, input, config, **kwargs)
+
+
+async def per_req_config_modifier(config: Dict, request: Request) -> Dict:
+    """Modify the config for each request."""
+    user = await get_current_active_user_from_request(request)
+    config["configurable"] = {}
+    # Attention: Make sure that the user ID is over-ridden for each request.
+    # We should not be accepting a user ID from the user in this case!
+    config["configurable"]["user_id"] = user.username
+    return config
+
+
+vectorstore = Chroma(
+    collection_name="some_collection",
+    embedding_function=OpenAIEmbeddings(),
+)
+
+vectorstore.add_documents(
+    [
+        Document(
+            page_content="cats like cheese",
+            metadata={"owner_id": "alice"},
+        ),
+        Document(
+            page_content="cats like mice",
+            metadata={"owner_id": "alice"},
+        ),
+        Document(
+            page_content="dogs like sticks",
+            metadata={"owner_id": "john"},
+        ),
+        Document(
+            page_content="my favorite food is cheese",
+            metadata={"owner_id": "john"},
+        ),
+        Document(
+            page_content="i like walks by the ocean",
+            metadata={"owner_id": "john"},
+        ),
+        Document(
+            page_content="dogs like grass",
+            metadata={"owner_id": "bob"},
+        ),
+    ]
+)
+
+per_user_retriever = PerUserVectorstore(
+    user_id=None,  # Placeholder ID that will be replaced by the per_req_config_modifier
+    vectorstore=vectorstore,
+).configurable_fields(
+    # Attention: Make sure to override the user ID for each request in the
+    # per_req_config_modifier. This should not be client configurable.
+    user_id=ConfigurableField(
+        id="user_id",
+        name="User ID",
+        description="The user ID to use for the retriever.",
+    )
+)
+
+add_routes(
+    app,
+    per_user_retriever,
+    per_req_config_modifier=per_req_config_modifier,
+    enabled_endpoints=["invoke"],
+)
+
+
+if __name__ == "__main__":
+    import uvicorn
+
+    uvicorn.run(app, host="localhost", port=8000)

examples/chat_with_persistence/client.ipynb

@@ -0,0 +1,277 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Chat History\n",
+    "\n",
+    "An example of a client interacting with a chatbot where message history is persisted on the backend."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "import uuid\n",
+    "from langserve import RemoteRunnable\n",
+    "\n",
+    "chat = RemoteRunnable(\"http://localhost:8000/\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Let's create a prompt composed of a system message and a human message."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "session_id = str(uuid.uuid4())"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "AIMessage(content=\" Hello Eugene! My name is Claude. It's nice to meet another cat lover.\")"
+      ]
+     },
+     "execution_count": 3,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "chat.invoke({\"human_input\": \"my name is eugene. i like cats. what is your name?\"}, {'configurable': { 'session_id': session_id } })"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "AIMessage(content=' You told me your name is Eugene.')"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "chat.invoke({\"human_input\": \"what was my name?\"}, {'configurable': { 'session_id': session_id } })"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "AIMessage(content=' You said you like cats.')"
+      ]
+     },
+     "execution_count": 5,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "chat.invoke({\"human_input\": \"What animal do i like?\"}, {'configurable': { 'session_id': session_id } })"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      " Sure\n",
+      ",\n",
+      " I\n",
+      "'d\n",
+      " be\n",
+      " happy\n",
+      " to\n",
+      " count\n",
+      " to\n",
+      " 10\n",
+      ":\n",
+      "\n",
+      "\n",
+      "1\n",
+      ",\n",
+      " 2\n",
+      ",\n",
+      " 3\n",
+      ",\n",
+      " 4\n",
+      ",\n",
+      " 5\n",
+      ",\n",
+      " 6\n",
+      ",\n",
+      " 7\n",
+      ",\n",
+      " 8\n",
+      ",\n",
+      " 9\n",
+      ",\n",
+      " 10\n"
+     ]
+    }
+   ],
+   "source": [
+    "for chunk in chat.stream({'human_input': \"Can you count till 10?\"},  {'configurable': { 'session_id': session_id } }):\n",
+    "    print()\n",
+    "    print(chunk.content, end='', flush=True)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\u001b[1;39m[\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"type\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"human\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"data\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
+      "      \u001b[0m\u001b[34;1m\"content\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"my name is eugene. i like cats. what is your name?\"\u001b[0m\u001b[1;39m,\n",
+      "      \u001b[0m\u001b[34;1m\"additional_kwargs\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{}\u001b[0m\u001b[1;39m,\n",
+      "      \u001b[0m\u001b[34;1m\"type\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"human\"\u001b[0m\u001b[1;39m,\n",
+      "      \u001b[0m\u001b[34;1m\"example\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;39mfalse\u001b[0m\u001b[1;39m\n",
+      "    \u001b[1;39m}\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"type\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"ai\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"data\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
+      "      \u001b[0m\u001b[34;1m\"content\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\" Hello Eugene! My name is Claude. It's nice to meet another cat lover.\"\u001b[0m\u001b[1;39m,\n",
+      "      \u001b[0m\u001b[34;1m\"additional_kwargs\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{}\u001b[0m\u001b[1;39m,\n",
+      "      \u001b[0m\u001b[34;1m\"type\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"ai\"\u001b[0m\u001b[1;39m,\n",
+      "      \u001b[0m\u001b[34;1m\"example\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;39mfalse\u001b[0m\u001b[1;39m\n",
+      "    \u001b[1;39m}\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"type\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"human\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"data\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
+      "      \u001b[0m\u001b[34;1m\"content\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"what was my name?\"\u001b[0m\u001b[1;39m,\n",
+      "      \u001b[0m\u001b[34;1m\"additional_kwargs\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{}\u001b[0m\u001b[1;39m,\n",
+      "      \u001b[0m\u001b[34;1m\"type\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"human\"\u001b[0m\u001b[1;39m,\n",
+      "      \u001b[0m\u001b[34;1m\"example\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;39mfalse\u001b[0m\u001b[1;39m\n",
+      "    \u001b[1;39m}\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"type\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"ai\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"data\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
+      "      \u001b[0m\u001b[34;1m\"content\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\" You told me your name is Eugene.\"\u001b[0m\u001b[1;39m,\n",
+      "      \u001b[0m\u001b[34;1m\"additional_kwargs\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{}\u001b[0m\u001b[1;39m,\n",
+      "      \u001b[0m\u001b[34;1m\"type\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"ai\"\u001b[0m\u001b[1;39m,\n",
+      "      \u001b[0m\u001b[34;1m\"example\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;39mfalse\u001b[0m\u001b[1;39m\n",
+      "    \u001b[1;39m}\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"type\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"human\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"data\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
+      "      \u001b[0m\u001b[34;1m\"content\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"What animal do i like?\"\u001b[0m\u001b[1;39m,\n",
+      "      \u001b[0m\u001b[34;1m\"additional_kwargs\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{}\u001b[0m\u001b[1;39m,\n",
+      "      \u001b[0m\u001b[34;1m\"type\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"human\"\u001b[0m\u001b[1;39m,\n",
+      "      \u001b[0m\u001b[34;1m\"example\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;39mfalse\u001b[0m\u001b[1;39m\n",
+      "    \u001b[1;39m}\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"type\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"ai\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"data\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
+      "      \u001b[0m\u001b[34;1m\"content\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\" You said you like cats.\"\u001b[0m\u001b[1;39m,\n",
+      "      \u001b[0m\u001b[34;1m\"additional_kwargs\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{}\u001b[0m\u001b[1;39m,\n",
+      "      \u001b[0m\u001b[34;1m\"type\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"ai\"\u001b[0m\u001b[1;39m,\n",
+      "      \u001b[0m\u001b[34;1m\"example\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;39mfalse\u001b[0m\u001b[1;39m\n",
+      "    \u001b[1;39m}\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"type\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"human\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"data\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
+      "      \u001b[0m\u001b[34;1m\"content\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"Can you count till 10?\"\u001b[0m\u001b[1;39m,\n",
+      "      \u001b[0m\u001b[34;1m\"additional_kwargs\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{}\u001b[0m\u001b[1;39m,\n",
+      "      \u001b[0m\u001b[34;1m\"type\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"human\"\u001b[0m\u001b[1;39m,\n",
+      "      \u001b[0m\u001b[34;1m\"example\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;39mfalse\u001b[0m\u001b[1;39m\n",
+      "    \u001b[1;39m}\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"type\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"AIMessageChunk\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"data\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
+      "      \u001b[0m\u001b[34;1m\"content\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\" Sure, I'd be happy to count to 10:\\n\\n1, 2, 3, 4, 5, 6, 7, 8, 9, 10\"\u001b[0m\u001b[1;39m,\n",
+      "      \u001b[0m\u001b[34;1m\"additional_kwargs\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{}\u001b[0m\u001b[1;39m,\n",
+      "      \u001b[0m\u001b[34;1m\"type\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"AIMessageChunk\"\u001b[0m\u001b[1;39m,\n",
+      "      \u001b[0m\u001b[34;1m\"example\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;39mfalse\u001b[0m\u001b[1;39m\n",
+      "    \u001b[1;39m}\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m\n",
+      "\u001b[1;39m]\u001b[0m\n"
+     ]
+    }
+   ],
+   "source": [
+    "!cat chat_histories/c7a327f3-5578-4fb7-a8f2-3082d7cb58cc.json | jq ."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.6"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

examples/chat_with_persistence/server.py

@@ -0,0 +1,113 @@
+#!/usr/bin/env python
+"""Example of a chat server with persistence handled on the backend.
+
+For simplicity, we're using file storage here -- to avoid the need to set up
+a database. This is obviously not a good idea for a production environment,
+but will help us to demonstrate the RunnableWithMessageHistory interface.
+
+We'll use cookies to identify the user and/or session. This will help illustrate how to
+fetch configuration from the request.
+"""
+import re
+from pathlib import Path
+from typing import Callable, Union
+
+from fastapi import FastAPI, HTTPException
+from langchain.chat_models import ChatAnthropic
+from langchain.memory import FileChatMessageHistory
+from langchain_core.chat_history import BaseChatMessageHistory
+from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
+from langchain_core.runnables.history import RunnableWithMessageHistory
+
+from langserve import add_routes
+from langserve.pydantic_v1 import BaseModel, Field
+
+
+def _is_valid_identifier(value: str) -> bool:
+    """Check if the session ID is in a valid format."""
+    # Use a regular expression to match the allowed characters
+    valid_characters = re.compile(r"^[a-zA-Z0-9-_]+$")
+    return bool(valid_characters.match(value))
+
+
+def create_session_factory(
+    base_dir: Union[str, Path],
+) -> Callable[[str], BaseChatMessageHistory]:
+    """Create a session ID factory that creates session IDs from a base dir.
+
+    Args:
+        base_dir: Base directory to use for storing the chat histories.
+
+    Returns:
+        A session ID factory that creates session IDs from a base path.
+    """
+    base_dir_ = Path(base_dir) if isinstance(base_dir, str) else base_dir
+    if not base_dir_.exists():
+        base_dir_.mkdir(parents=True)
+
+    def get_chat_history(session_id: str) -> FileChatMessageHistory:
+        """Get a chat history from a session ID."""
+        if not _is_valid_identifier(session_id):
+            raise HTTPException(
+                status_code=400,
+                detail=f"Session ID `{session_id}` is not in a valid format. "
+                "Session ID must only contain alphanumeric characters, "
+                "hyphens, and underscores.",
+            )
+        file_path = base_dir_ / f"{session_id}.json"
+        return FileChatMessageHistory(str(file_path))
+
+    return get_chat_history
+
+
+app = FastAPI(
+    title="LangChain Server",
+    version="1.0",
+    description="Spin up a simple api server using Langchain's Runnable interfaces",
+)
+
+
+# Declare a chain
+prompt = ChatPromptTemplate.from_messages(
+    [
+        ("system", "You're an assistant by the name of Bob."),
+        MessagesPlaceholder(variable_name="history"),
+        ("human", "{human_input}"),
+    ]
+)
+
+chain = prompt | ChatAnthropic(model="claude-2")
+
+
+class InputChat(BaseModel):
+    """Input for the chat endpoint."""
+
+    # The field extra defines a chat widget.
+    # As of 2024-02-05, this chat widget is not fully supported.
+    # It's included in documentation to show how it should be specified, but
+    # will not work until the widget is fully supported for history persistence
+    # on the backend.
+    human_input: str = Field(
+        ...,
+        description="The human input to the chat system.",
+        extra={"widget": {"type": "chat", "input": "human_input"}},
+    )
+
+
+chain_with_history = RunnableWithMessageHistory(
+    chain,
+    create_session_factory("chat_histories"),
+    input_messages_key="human_input",
+    history_messages_key="history",
+).with_types(input_type=InputChat)
+
+
+add_routes(
+    app,
+    chain_with_history,
+)
+
+if __name__ == "__main__":
+    import uvicorn
+
+    uvicorn.run(app, host="localhost", port=8000)

examples/chat_with_persistence_and_user/client.ipynb

@@ -0,0 +1,359 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Chat History\n",
+    "\n",
+    "Here we'll be interacting with a server that's exposing a chat bot with message history being persisted on the backend."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "import uuid\n",
+    "from langserve import RemoteRunnable\n",
+    "\n",
+    "conversation_id = str(uuid.uuid4())\n",
+    "chat = RemoteRunnable(\"http://localhost:8000/\", cookies={\"user_id\": \"eugene\"})"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Let's create a prompt composed of a system message and a human message."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "AIMessage(content=\"Hello Eugene! I'm Bob, your virtual assistant. How can I assist you today?\")"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "chat.invoke({\"human_input\": \"my name is eugene. what is your name?\"}, {'configurable': { 'conversation_id': conversation_id } })"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "AIMessage(content='Your name is Eugene. Is there something specific you would like assistance with, Eugene?')"
+      ]
+     },
+     "execution_count": 5,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "chat.invoke({\"human_input\": \"what was my name?\"}, {'configurable': { 'conversation_id': conversation_id } })"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Use different user but same conversation id"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "chat = RemoteRunnable(\"http://localhost:8000/\", cookies={\"user_id\": \"nuno\"})"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "AIMessage(content=\"I apologize, but I don't have access to personal information about users. As an AI assistant, I prioritize user privacy and data protection.\")"
+      ]
+     },
+     "execution_count": 7,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "chat.invoke({\"human_input\": \"what was my name?\"}, {'configurable': { 'conversation_id': conversation_id }})"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\n",
+      "Of\n",
+      " course\n",
+      "!\n",
+      " Here\n",
+      " you\n",
+      " go\n",
+      ":\n",
+      "\n",
+      "\n",
+      "1\n",
+      ",\n",
+      " \n",
+      "2\n",
+      ",\n",
+      " \n",
+      "3\n",
+      ",\n",
+      " \n",
+      "4\n",
+      ",\n",
+      " \n",
+      "5\n",
+      ",\n",
+      " \n",
+      "6\n",
+      ",\n",
+      " \n",
+      "7\n",
+      ",\n",
+      " \n",
+      "8\n",
+      ",\n",
+      " \n",
+      "9\n",
+      ",\n",
+      " \n",
+      "10\n",
+      ".\n"
+     ]
+    }
+   ],
+   "source": [
+    "for chunk in chat.stream({'human_input': \"Can you count till 10?\"},  {'configurable': { 'conversation_id': conversation_id } }):\n",
+    "    print()\n",
+    "    print(chunk.content, end='', flush=True)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "'cd8e5a55-0295-41cd-a885-775e0403fd25'"
+      ]
+     },
+     "execution_count": 11,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "conversation_id"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\u001b[01;34mchat_histories/\u001b[0m\n",
+      "├── \u001b[01;34meugene\u001b[0m\n",
+      "│   └── cd8e5a55-0295-41cd-a885-775e0403fd25.json\n",
+      "└── \u001b[01;34mnuno\u001b[0m\n",
+      "    └── cd8e5a55-0295-41cd-a885-775e0403fd25.json\n",
+      "\n",
+      "2 directories, 2 files\n"
+     ]
+    }
+   ],
+   "source": [
+    "!tree chat_histories/"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 17,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\u001b[1;39m[\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"type\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"human\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"data\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
+      "      \u001b[0m\u001b[34;1m\"content\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"my name is eugene. what is your name?\"\u001b[0m\u001b[1;39m,\n",
+      "      \u001b[0m\u001b[34;1m\"additional_kwargs\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{}\u001b[0m\u001b[1;39m,\n",
+      "      \u001b[0m\u001b[34;1m\"type\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"human\"\u001b[0m\u001b[1;39m,\n",
+      "      \u001b[0m\u001b[34;1m\"example\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;39mfalse\u001b[0m\u001b[1;39m\n",
+      "    \u001b[1;39m}\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"type\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"ai\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"data\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
+      "      \u001b[0m\u001b[34;1m\"content\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"Hello Eugene! I'm Bob, your virtual assistant. How can I assist you today?\"\u001b[0m\u001b[1;39m,\n",
+      "      \u001b[0m\u001b[34;1m\"additional_kwargs\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{}\u001b[0m\u001b[1;39m,\n",
+      "      \u001b[0m\u001b[34;1m\"type\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"ai\"\u001b[0m\u001b[1;39m,\n",
+      "      \u001b[0m\u001b[34;1m\"example\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;39mfalse\u001b[0m\u001b[1;39m\n",
+      "    \u001b[1;39m}\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"type\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"human\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"data\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
+      "      \u001b[0m\u001b[34;1m\"content\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"what was my name?\"\u001b[0m\u001b[1;39m,\n",
+      "      \u001b[0m\u001b[34;1m\"additional_kwargs\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{}\u001b[0m\u001b[1;39m,\n",
+      "      \u001b[0m\u001b[34;1m\"type\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"human\"\u001b[0m\u001b[1;39m,\n",
+      "      \u001b[0m\u001b[34;1m\"example\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;39mfalse\u001b[0m\u001b[1;39m\n",
+      "    \u001b[1;39m}\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"type\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"ai\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"data\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
+      "      \u001b[0m\u001b[34;1m\"content\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"Your name is Eugene. Is there something specific you would like assistance with, Eugene?\"\u001b[0m\u001b[1;39m,\n",
+      "      \u001b[0m\u001b[34;1m\"additional_kwargs\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{}\u001b[0m\u001b[1;39m,\n",
+      "      \u001b[0m\u001b[34;1m\"type\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"ai\"\u001b[0m\u001b[1;39m,\n",
+      "      \u001b[0m\u001b[34;1m\"example\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;39mfalse\u001b[0m\u001b[1;39m\n",
+      "    \u001b[1;39m}\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m\n",
+      "\u001b[1;39m]\u001b[0m\n"
+     ]
+    }
+   ],
+   "source": [
+    "!cat chat_histories/eugene/cd8e5a55-0295-41cd-a885-775e0403fd25.json | jq ."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 18,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\u001b[1;39m[\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"type\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"human\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"data\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
+      "      \u001b[0m\u001b[34;1m\"content\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"what was my name?\"\u001b[0m\u001b[1;39m,\n",
+      "      \u001b[0m\u001b[34;1m\"additional_kwargs\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{}\u001b[0m\u001b[1;39m,\n",
+      "      \u001b[0m\u001b[34;1m\"type\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"human\"\u001b[0m\u001b[1;39m,\n",
+      "      \u001b[0m\u001b[34;1m\"example\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;39mfalse\u001b[0m\u001b[1;39m\n",
+      "    \u001b[1;39m}\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"type\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"ai\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"data\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
+      "      \u001b[0m\u001b[34;1m\"content\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"I apologize, but I don't have access to personal information about users. As an AI assistant, I prioritize user privacy and data protection.\"\u001b[0m\u001b[1;39m,\n",
+      "      \u001b[0m\u001b[34;1m\"additional_kwargs\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{}\u001b[0m\u001b[1;39m,\n",
+      "      \u001b[0m\u001b[34;1m\"type\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"ai\"\u001b[0m\u001b[1;39m,\n",
+      "      \u001b[0m\u001b[34;1m\"example\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;39mfalse\u001b[0m\u001b[1;39m\n",
+      "    \u001b[1;39m}\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"type\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"human\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"data\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
+      "      \u001b[0m\u001b[34;1m\"content\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"Can you count till 10?\"\u001b[0m\u001b[1;39m,\n",
+      "      \u001b[0m\u001b[34;1m\"additional_kwargs\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{}\u001b[0m\u001b[1;39m,\n",
+      "      \u001b[0m\u001b[34;1m\"type\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"human\"\u001b[0m\u001b[1;39m,\n",
+      "      \u001b[0m\u001b[34;1m\"example\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;39mfalse\u001b[0m\u001b[1;39m\n",
+      "    \u001b[1;39m}\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m,\n",
+      "  \u001b[1;39m{\n",
+      "    \u001b[0m\u001b[34;1m\"type\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"AIMessageChunk\"\u001b[0m\u001b[1;39m,\n",
+      "    \u001b[0m\u001b[34;1m\"data\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{\n",
+      "      \u001b[0m\u001b[34;1m\"content\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"Of course! Here you go:\\n\\n1, 2, 3, 4, 5, 6, 7, 8, 9, 10.\"\u001b[0m\u001b[1;39m,\n",
+      "      \u001b[0m\u001b[34;1m\"additional_kwargs\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m{}\u001b[0m\u001b[1;39m,\n",
+      "      \u001b[0m\u001b[34;1m\"type\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"AIMessageChunk\"\u001b[0m\u001b[1;39m,\n",
+      "      \u001b[0m\u001b[34;1m\"example\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;39mfalse\u001b[0m\u001b[1;39m\n",
+      "    \u001b[1;39m}\u001b[0m\u001b[1;39m\n",
+      "  \u001b[1;39m}\u001b[0m\u001b[1;39m\n",
+      "\u001b[1;39m]\u001b[0m\n"
+     ]
+    }
+   ],
+   "source": [
+    "!cat chat_histories/nuno/cd8e5a55-0295-41cd-a885-775e0403fd25.json | jq ."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.6"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

examples/chat_with_persistence_and_user/server.py

@@ -0,0 +1,182 @@
+#!/usr/bin/env python
+"""Example of a chat server with persistence handled on the backend.
+
+For simplicity, we're using file storage here -- to avoid the need to set up
+a database. This is obviously not a good idea for a production environment,
+but will help us to demonstrate the RunnableWithMessageHistory interface.
+
+We'll use cookies to identify the user. This will help illustrate how to
+fetch configuration from the request.
+"""
+import re
+from pathlib import Path
+from typing import Any, Callable, Dict, Union
+
+from fastapi import FastAPI, HTTPException, Request
+from langchain.chat_models import ChatOpenAI
+from langchain.memory import FileChatMessageHistory
+from langchain.schema.runnable.utils import ConfigurableFieldSpec
+from langchain_core import __version__
+from langchain_core.chat_history import BaseChatMessageHistory
+from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
+from langchain_core.runnables.history import RunnableWithMessageHistory
+from typing_extensions import TypedDict
+
+from langserve import add_routes
+
+# Define the minimum required version as (0, 1, 0)
+# Earlier versions did not allow specifying custom config fields in
+# RunnableWithMessageHistory.
+MIN_VERSION_LANGCHAIN_CORE = (0, 1, 0)
+
+# Split the version string by "." and convert to integers
+LANGCHAIN_CORE_VERSION = tuple(map(int, __version__.split(".")))
+
+if LANGCHAIN_CORE_VERSION < MIN_VERSION_LANGCHAIN_CORE:
+    raise RuntimeError(
+        f"Minimum required version of langchain-core is {MIN_VERSION_LANGCHAIN_CORE}, "
+        f"but found {LANGCHAIN_CORE_VERSION}"
+    )
+
+
+def _is_valid_identifier(value: str) -> bool:
+    """Check if the value is a valid identifier."""
+    # Use a regular expression to match the allowed characters
+    valid_characters = re.compile(r"^[a-zA-Z0-9-_]+$")
+    return bool(valid_characters.match(value))
+
+
+def create_session_factory(
+    base_dir: Union[str, Path],
+) -> Callable[[str], BaseChatMessageHistory]:
+    """Create a factory that can retrieve chat histories.
+
+    The chat histories are keyed by user ID and conversation ID.
+
+    Args:
+        base_dir: Base directory to use for storing the chat histories.
+
+    Returns:
+        A factory that can retrieve chat histories keyed by user ID and conversation ID.
+    """
+    base_dir_ = Path(base_dir) if isinstance(base_dir, str) else base_dir
+    if not base_dir_.exists():
+        base_dir_.mkdir(parents=True)
+
+    def get_chat_history(user_id: str, conversation_id: str) -> FileChatMessageHistory:
+        """Get a chat history from a user id and conversation id."""
+        if not _is_valid_identifier(user_id):
+            raise ValueError(
+                f"User ID {user_id} is not in a valid format. "
+                "User ID must only contain alphanumeric characters, "
+                "hyphens, and underscores."
+                "Please include a valid cookie in the request headers called 'user-id'."
+            )
+        if not _is_valid_identifier(conversation_id):
+            raise ValueError(
+                f"Conversation ID {conversation_id} is not in a valid format. "
+                "Conversation ID must only contain alphanumeric characters, "
+                "hyphens, and underscores. Please provide a valid conversation id "
+                "via config. For example, "
+                "chain.invoke(.., {'configurable': {'conversation_id': '123'}})"
+            )
+
+        user_dir = base_dir_ / user_id
+        if not user_dir.exists():
+            user_dir.mkdir(parents=True)
+        file_path = user_dir / f"{conversation_id}.json"
+        return FileChatMessageHistory(str(file_path))
+
+    return get_chat_history
+
+
+app = FastAPI(
+    title="LangChain Server",
+    version="1.0",
+    description="Spin up a simple api server using Langchain's Runnable interfaces",
+)
+
+
+def _per_request_config_modifier(
+    config: Dict[str, Any], request: Request
+) -> Dict[str, Any]:
+    """Update the config"""
+    config = config.copy()
+    configurable = config.get("configurable", {})
+    # Look for a cookie named "user_id"
+    user_id = request.cookies.get("user_id", None)
+
+    if user_id is None:
+        raise HTTPException(
+            status_code=400,
+            detail="No user id found. Please set a cookie named 'user_id'.",
+        )
+
+    configurable["user_id"] = user_id
+    config["configurable"] = configurable
+    return config
+
+
+# Declare a chain
+prompt = ChatPromptTemplate.from_messages(
+    [
+        ("system", "You're an assistant by the name of Bob."),
+        MessagesPlaceholder(variable_name="history"),
+        ("human", "{human_input}"),
+    ]
+)
+
+chain = prompt | ChatOpenAI()
+
+
+class InputChat(TypedDict):
+    """Input for the chat endpoint."""
+
+    human_input: str
+    """Human input"""
+
+
+chain_with_history = RunnableWithMessageHistory(
+    chain,
+    create_session_factory("chat_histories"),
+    input_messages_key="human_input",
+    history_messages_key="history",
+    history_factory_config=[
+        ConfigurableFieldSpec(
+            id="user_id",
+            annotation=str,
+            name="User ID",
+            description="Unique identifier for the user.",
+            default="",
+            is_shared=True,
+        ),
+        ConfigurableFieldSpec(
+            id="conversation_id",
+            annotation=str,
+            name="Conversation ID",
+            description="Unique identifier for the conversation.",
+            default="",
+            is_shared=True,
+        ),
+    ],
+).with_types(input_type=InputChat)
+
+
+add_routes(
+    app,
+    chain_with_history,
+    per_req_config_modifier=_per_request_config_modifier,
+    # Disable playground and batch
+    # 1) Playground we're passing information via headers, which is not supported via
+    #    the playground right now.
+    # 2) Disable batch to avoid users being confused. Batch will work fine
+    #    as long as users invoke it with multiple configs appropriately, but
+    #    without validation users are likely going to forget to do that.
+    #    In addition, there's likely little sense in support batch for a chatbot.
+    disabled_endpoints=["playground", "batch"],
+)
+
+if __name__ == "__main__":
+    import uvicorn
+
+    uvicorn.run(app, host="localhost", port=8000)

examples/configurable_agent_executor/client.ipynb

@@ -0,0 +1,770 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Client\n",
+    "\n",
+    "Demo of a client interacting with a custom runnable executor that supports configuration.\n",
+    "\n",
+    "This server does not support invoke or batch! only stream and astream log! (see backend code.)\n",
+    "\n",
+    "The underlying backend code is just a demo in this case -- it's working around an existing bug, but uses \n",
+    "the opportunity to show how to create custom runnables."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "You can interact with this via API directly"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "event: metadata\n",
+      "data: {\"run_id\": \"5e6ce60a-95c4-4fe5-8c4a-ec1d347afd83\"}\n",
+      "\n",
+      "event: data\n",
+      "data: {\"actions\":[{\"tool\":\"get_eugene_thoughts\",\"tool_input\":{\"query\":\"cats\"},\"log\":\"\\nInvoking: `get_eugene_thoughts` with `{'query': 'cats'}`\\n\\n\\n\",\"type\":\"AgentActionMessageLog\",\"message_log\":[{\"content\":\"\",\"additional_kwargs\":{\"function_call\":{\"name\":\"get_eugene_thoughts\",\"arguments\":\"{\\n  \\\"query\\\": \\\"cats\\\"\\n}\"}},\"type\":\"ai\",\"example\":false}]}],\"messages\":[{\"content\":\"\",\"additional_kwargs\":{\"function_call\":{\"name\":\"get_eugene_thoughts\",\"arguments\":\"{\\n  \\\"query\\\": \\\"cats\\\"\\n}\"}},\"type\":\"ai\",\"example\":false}]}\n",
+      "\n",
+      "event: data\n",
+      "data: {\"steps\":[{\"action\":{\"tool\":\"get_eugene_thoughts\",\"tool_input\":{\"query\":\"cats\"},\"log\":\"\\nInvoking: `get_eugene_thoughts` with `{'query': 'cats'}`\\n\\n\\n\",\"type\":\"AgentActionMessageLog\",\"message_log\":[{\"content\":\"\",\"additional_kwargs\":{\"function_call\":{\"name\":\"get_eugene_thoughts\",\"arguments\":\"{\\n  \\\"query\\\": \\\"cats\\\"\\n}\"}},\"type\":\"ai\",\"example\":false}]},\"observation\":[{\"page_content\":\"cats like fish\",\"metadata\":{},\"type\":\"Document\"},{\"page_content\":\"dogs like sticks\",\"metadata\":{},\"type\":\"Document\"}]}],\"messages\":[{\"content\":\"[Document(page_content='cats like fish'), Document(page_content='dogs like sticks')]\",\"additional_kwargs\":{},\"type\":\"function\",\"name\":\"get_eugene_thoughts\"}]}\n",
+      "\n",
+      "event: data\n",
+      "data: {\"output\":\"Eugene thinks that cats like fish.\",\"messages\":[{\"content\":\"Eugene thinks that cats like fish.\",\"additional_kwargs\":{},\"type\":\"ai\",\"example\":false}]}\n",
+      "\n",
+      "event: end\n",
+      "\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "import requests\n",
+    "\n",
+    "inputs = {\"input\": {\"input\": \"what does eugene think of cats?\"}}\n",
+    "response = requests.post(\"http://localhost:8000/stream\", json=inputs)\n",
+    "\n",
+    "print(response.text)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "You can also interact with this via the RemoteRunnable interface (to use in other chains)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "from langserve import RemoteRunnable\n",
+    "\n",
+    "remote_runnable = RemoteRunnable(\"http://localhost:8000/\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Remote runnable has the same interface as local runnables"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "{'output': 'Hello! How can I assist you today?', 'messages': [AIMessage(content='Hello! How can I assist you today?')]}\n"
+     ]
+    }
+   ],
+   "source": [
+    "async for chunk in remote_runnable.astream({\"input\": \"hi!\"}):\n",
+    "    print(chunk)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "RunLogPatch({'op': 'replace',\n",
+      "  'path': '',\n",
+      "  'value': {'final_output': None,\n",
+      "            'id': '9f415b49-ba69-4fdf-9b9c-5ccc1805487f',\n",
+      "            'logs': {},\n",
+      "            'streamed_output': []}})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/RunnableSequence',\n",
+      "  'value': {'end_time': None,\n",
+      "            'final_output': None,\n",
+      "            'id': '95a0ba72-9511-4fff-8b8c-410e7108ee60',\n",
+      "            'metadata': {'__langserve_endpoint': 'stream_log',\n",
+      "                         '__langserve_version': '0.0.37',\n",
+      "                         '__useragent': 'python-httpx/0.25.2'},\n",
+      "            'name': 'RunnableSequence',\n",
+      "            'start_time': '2024-01-06T03:12:42.213+00:00',\n",
+      "            'streamed_output': [],\n",
+      "            'streamed_output_str': [],\n",
+      "            'tags': [],\n",
+      "            'type': 'chain'}})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/RunnableParallel<input,agent_scratchpad>',\n",
+      "  'value': {'end_time': None,\n",
+      "            'final_output': None,\n",
+      "            'id': 'a0afa5a6-a408-4a2d-83f8-8e4c8193f963',\n",
+      "            'metadata': {'__langserve_endpoint': 'stream_log',\n",
+      "                         '__langserve_version': '0.0.37',\n",
+      "                         '__useragent': 'python-httpx/0.25.2'},\n",
+      "            'name': 'RunnableParallel<input,agent_scratchpad>',\n",
+      "            'start_time': '2024-01-06T03:12:42.214+00:00',\n",
+      "            'streamed_output': [],\n",
+      "            'streamed_output_str': [],\n",
+      "            'tags': ['seq:step:1'],\n",
+      "            'type': 'chain'}})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/<lambda>',\n",
+      "  'value': {'end_time': None,\n",
+      "            'final_output': None,\n",
+      "            'id': '2aed7d13-2d0b-4c37-a4b3-d3f266bbf7e6',\n",
+      "            'metadata': {'__langserve_endpoint': 'stream_log',\n",
+      "                         '__langserve_version': '0.0.37',\n",
+      "                         '__useragent': 'python-httpx/0.25.2'},\n",
+      "            'name': '<lambda>',\n",
+      "            'start_time': '2024-01-06T03:12:42.215+00:00',\n",
+      "            'streamed_output': [],\n",
+      "            'streamed_output_str': [],\n",
+      "            'tags': ['map:key:input'],\n",
+      "            'type': 'chain'}})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/<lambda>:2',\n",
+      "  'value': {'end_time': None,\n",
+      "            'final_output': None,\n",
+      "            'id': '9328b5cf-fd7b-48fe-affb-2973643189e9',\n",
+      "            'metadata': {'__langserve_endpoint': 'stream_log',\n",
+      "                         '__langserve_version': '0.0.37',\n",
+      "                         '__useragent': 'python-httpx/0.25.2'},\n",
+      "            'name': '<lambda>',\n",
+      "            'start_time': '2024-01-06T03:12:42.215+00:00',\n",
+      "            'streamed_output': [],\n",
+      "            'streamed_output_str': [],\n",
+      "            'tags': ['map:key:agent_scratchpad'],\n",
+      "            'type': 'chain'}})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/<lambda>/final_output',\n",
+      "  'value': {'output': 'what does eugene think about cats?'}},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/<lambda>/end_time',\n",
+      "  'value': '2024-01-06T03:12:42.217+00:00'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/<lambda>:2/final_output',\n",
+      "  'value': {'output': []}},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/<lambda>:2/end_time',\n",
+      "  'value': '2024-01-06T03:12:42.217+00:00'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/RunnableParallel<input,agent_scratchpad>/final_output',\n",
+      "  'value': {'agent_scratchpad': [],\n",
+      "            'input': 'what does eugene think about cats?'}},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/RunnableParallel<input,agent_scratchpad>/end_time',\n",
+      "  'value': '2024-01-06T03:12:42.218+00:00'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatPromptTemplate',\n",
+      "  'value': {'end_time': None,\n",
+      "            'final_output': None,\n",
+      "            'id': 'a08bec71-bf7f-46c5-b082-14f7bff421cd',\n",
+      "            'metadata': {'__langserve_endpoint': 'stream_log',\n",
+      "                         '__langserve_version': '0.0.37',\n",
+      "                         '__useragent': 'python-httpx/0.25.2'},\n",
+      "            'name': 'ChatPromptTemplate',\n",
+      "            'start_time': '2024-01-06T03:12:42.219+00:00',\n",
+      "            'streamed_output': [],\n",
+      "            'streamed_output_str': [],\n",
+      "            'tags': ['seq:step:2'],\n",
+      "            'type': 'prompt'}})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatPromptTemplate/final_output',\n",
+      "  'value': {'messages': [SystemMessage(content='You are a helpful assistant.'),\n",
+      "                         HumanMessage(content='what does eugene think about cats?')]}},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/ChatPromptTemplate/end_time',\n",
+      "  'value': '2024-01-06T03:12:42.219+00:00'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/LLM',\n",
+      "  'value': {'end_time': None,\n",
+      "            'final_output': None,\n",
+      "            'id': 'bf859261-467b-44b3-84fb-80d8c9e9dff2',\n",
+      "            'metadata': {'__langserve_endpoint': 'stream_log',\n",
+      "                         '__langserve_version': '0.0.37',\n",
+      "                         '__useragent': 'python-httpx/0.25.2'},\n",
+      "            'name': 'LLM',\n",
+      "            'start_time': '2024-01-06T03:12:42.221+00:00',\n",
+      "            'streamed_output': [],\n",
+      "            'streamed_output_str': [],\n",
+      "            'tags': ['seq:step:3'],\n",
+      "            'type': 'llm'}})\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/LLM/streamed_output_str/-', 'value': ''},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/LLM/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='', additional_kwargs={'function_call': {'name': 'get_eugene_thoughts', 'arguments': ''}})})\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/LLM/streamed_output_str/-', 'value': ''},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/LLM/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='', additional_kwargs={'function_call': {'arguments': '{\\n'}})})\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/LLM/streamed_output_str/-', 'value': ''},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/LLM/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='', additional_kwargs={'function_call': {'arguments': ' '}})})\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/LLM/streamed_output_str/-', 'value': ''},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/LLM/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='', additional_kwargs={'function_call': {'arguments': ' \"'}})})\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/LLM/streamed_output_str/-', 'value': ''},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/LLM/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='', additional_kwargs={'function_call': {'arguments': 'query'}})})\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/LLM/streamed_output_str/-', 'value': ''},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/LLM/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='', additional_kwargs={'function_call': {'arguments': '\":'}})})\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/LLM/streamed_output_str/-', 'value': ''},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/LLM/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='', additional_kwargs={'function_call': {'arguments': ' \"'}})})\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/LLM/streamed_output_str/-', 'value': ''},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/LLM/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='', additional_kwargs={'function_call': {'arguments': 'cats'}})})\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/LLM/streamed_output_str/-', 'value': ''},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/LLM/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='', additional_kwargs={'function_call': {'arguments': '\"\\n'}})})\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/LLM/streamed_output_str/-', 'value': ''},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/LLM/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='', additional_kwargs={'function_call': {'arguments': '}'}})})\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/LLM/streamed_output_str/-', 'value': ''},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/LLM/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='')})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/LLM/final_output',\n",
+      "  'value': LLMResult(generations=[[ChatGeneration(generation_info={'finish_reason': 'function_call'}, message=AIMessage(content='', additional_kwargs={'function_call': {'name': 'get_eugene_thoughts', 'arguments': '{\\n  \"query\": \"cats\"\\n}'}}))]], llm_output=None, run=None)},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/LLM/end_time',\n",
+      "  'value': '2024-01-06T03:12:42.806+00:00'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/OpenAIFunctionsAgentOutputParser',\n",
+      "  'value': {'end_time': None,\n",
+      "            'final_output': None,\n",
+      "            'id': 'd635ccbb-1db3-4515-b1ca-0c14752d361a',\n",
+      "            'metadata': {'__langserve_endpoint': 'stream_log',\n",
+      "                         '__langserve_version': '0.0.37',\n",
+      "                         '__useragent': 'python-httpx/0.25.2'},\n",
+      "            'name': 'OpenAIFunctionsAgentOutputParser',\n",
+      "            'start_time': '2024-01-06T03:12:42.807+00:00',\n",
+      "            'streamed_output': [],\n",
+      "            'streamed_output_str': [],\n",
+      "            'tags': ['seq:step:4'],\n",
+      "            'type': 'parser'}})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/OpenAIFunctionsAgentOutputParser/final_output',\n",
+      "  'value': AgentActionMessageLog(tool='get_eugene_thoughts', tool_input={'query': 'cats'}, log=\"\\nInvoking: `get_eugene_thoughts` with `{'query': 'cats'}`\\n\\n\\n\", message_log=[AIMessage(content='', additional_kwargs={'function_call': {'name': 'get_eugene_thoughts', 'arguments': '{\\n  \"query\": \"cats\"\\n}'}})])},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/OpenAIFunctionsAgentOutputParser/end_time',\n",
+      "  'value': '2024-01-06T03:12:42.809+00:00'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/RunnableSequence/final_output',\n",
+      "  'value': {'output': AgentActionMessageLog(tool='get_eugene_thoughts', tool_input={'query': 'cats'}, log=\"\\nInvoking: `get_eugene_thoughts` with `{'query': 'cats'}`\\n\\n\\n\", message_log=[AIMessage(content='', additional_kwargs={'function_call': {'name': 'get_eugene_thoughts', 'arguments': '{\\n  \"query\": \"cats\"\\n}'}})])}},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/RunnableSequence/end_time',\n",
+      "  'value': '2024-01-06T03:12:42.809+00:00'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/streamed_output/-',\n",
+      "  'value': {'actions': [AgentActionMessageLog(tool='get_eugene_thoughts', tool_input={'query': 'cats'}, log=\"\\nInvoking: `get_eugene_thoughts` with `{'query': 'cats'}`\\n\\n\\n\", message_log=[AIMessage(content='', additional_kwargs={'function_call': {'name': 'get_eugene_thoughts', 'arguments': '{\\n  \"query\": \"cats\"\\n}'}})])],\n",
+      "            'messages': [AIMessage(content='', additional_kwargs={'function_call': {'name': 'get_eugene_thoughts', 'arguments': '{\\n  \"query\": \"cats\"\\n}'}})]}},\n",
+      " {'op': 'replace',\n",
+      "  'path': '/final_output',\n",
+      "  'value': {'actions': [AgentActionMessageLog(tool='get_eugene_thoughts', tool_input={'query': 'cats'}, log=\"\\nInvoking: `get_eugene_thoughts` with `{'query': 'cats'}`\\n\\n\\n\", message_log=[AIMessage(content='', additional_kwargs={'function_call': {'name': 'get_eugene_thoughts', 'arguments': '{\\n  \"query\": \"cats\"\\n}'}})])],\n",
+      "            'messages': [AIMessage(content='', additional_kwargs={'function_call': {'name': 'get_eugene_thoughts', 'arguments': '{\\n  \"query\": \"cats\"\\n}'}})]}})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/get_eugene_thoughts',\n",
+      "  'value': {'end_time': None,\n",
+      "            'final_output': None,\n",
+      "            'id': '5eecdbe8-3374-4f0e-8ae8-d8a7e2394ae1',\n",
+      "            'metadata': {'__langserve_endpoint': 'stream_log',\n",
+      "                         '__langserve_version': '0.0.37',\n",
+      "                         '__useragent': 'python-httpx/0.25.2'},\n",
+      "            'name': 'get_eugene_thoughts',\n",
+      "            'start_time': '2024-01-06T03:12:42.810+00:00',\n",
+      "            'streamed_output': [],\n",
+      "            'streamed_output_str': [],\n",
+      "            'tags': [],\n",
+      "            'type': 'tool'}})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/get_eugene_thoughts/final_output',\n",
+      "  'value': {'output': \"[Document(page_content='cats like fish'), \"\n",
+      "                      \"Document(page_content='dogs like sticks')]\"}},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/get_eugene_thoughts/end_time',\n",
+      "  'value': '2024-01-06T03:12:43.095+00:00'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/streamed_output/-',\n",
+      "  'value': {'messages': [FunctionMessage(content=\"[Document(page_content='cats like fish'), Document(page_content='dogs like sticks')]\", name='get_eugene_thoughts')],\n",
+      "            'steps': [{'action': AgentActionMessageLog(tool='get_eugene_thoughts', tool_input={'query': 'cats'}, log=\"\\nInvoking: `get_eugene_thoughts` with `{'query': 'cats'}`\\n\\n\\n\", message_log=[AIMessage(content='', additional_kwargs={'function_call': {'name': 'get_eugene_thoughts', 'arguments': '{\\n  \"query\": \"cats\"\\n}'}})]),\n",
+      "                       'observation': [Document(page_content='cats like fish'),\n",
+      "                                       Document(page_content='dogs like sticks')]}]}},\n",
+      " {'op': 'add',\n",
+      "  'path': '/final_output/steps',\n",
+      "  'value': [{'action': AgentActionMessageLog(tool='get_eugene_thoughts', tool_input={'query': 'cats'}, log=\"\\nInvoking: `get_eugene_thoughts` with `{'query': 'cats'}`\\n\\n\\n\", message_log=[AIMessage(content='', additional_kwargs={'function_call': {'name': 'get_eugene_thoughts', 'arguments': '{\\n  \"query\": \"cats\"\\n}'}})]),\n",
+      "             'observation': [Document(page_content='cats like fish'),\n",
+      "                             Document(page_content='dogs like sticks')]}]},\n",
+      " {'op': 'add',\n",
+      "  'path': '/final_output/messages/1',\n",
+      "  'value': FunctionMessage(content=\"[Document(page_content='cats like fish'), Document(page_content='dogs like sticks')]\", name='get_eugene_thoughts')})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/RunnableSequence:2',\n",
+      "  'value': {'end_time': None,\n",
+      "            'final_output': None,\n",
+      "            'id': '89bf8195-df48-4fde-81b0-9e1982051caa',\n",
+      "            'metadata': {'__langserve_endpoint': 'stream_log',\n",
+      "                         '__langserve_version': '0.0.37',\n",
+      "                         '__useragent': 'python-httpx/0.25.2'},\n",
+      "            'name': 'RunnableSequence',\n",
+      "            'start_time': '2024-01-06T03:12:43.097+00:00',\n",
+      "            'streamed_output': [],\n",
+      "            'streamed_output_str': [],\n",
+      "            'tags': [],\n",
+      "            'type': 'chain'}})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/RunnableParallel<input,agent_scratchpad>:2',\n",
+      "  'value': {'end_time': None,\n",
+      "            'final_output': None,\n",
+      "            'id': '567c178a-4897-4865-af27-7122432df7bd',\n",
+      "            'metadata': {'__langserve_endpoint': 'stream_log',\n",
+      "                         '__langserve_version': '0.0.37',\n",
+      "                         '__useragent': 'python-httpx/0.25.2'},\n",
+      "            'name': 'RunnableParallel<input,agent_scratchpad>',\n",
+      "            'start_time': '2024-01-06T03:12:43.098+00:00',\n",
+      "            'streamed_output': [],\n",
+      "            'streamed_output_str': [],\n",
+      "            'tags': ['seq:step:1'],\n",
+      "            'type': 'chain'}})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/<lambda>:3',\n",
+      "  'value': {'end_time': None,\n",
+      "            'final_output': None,\n",
+      "            'id': '03530072-e5b2-4d7e-b875-f77e3d47481b',\n",
+      "            'metadata': {'__langserve_endpoint': 'stream_log',\n",
+      "                         '__langserve_version': '0.0.37',\n",
+      "                         '__useragent': 'python-httpx/0.25.2'},\n",
+      "            'name': '<lambda>',\n",
+      "            'start_time': '2024-01-06T03:12:43.099+00:00',\n",
+      "            'streamed_output': [],\n",
+      "            'streamed_output_str': [],\n",
+      "            'tags': ['map:key:input'],\n",
+      "            'type': 'chain'}})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/<lambda>:4',\n",
+      "  'value': {'end_time': None,\n",
+      "            'final_output': None,\n",
+      "            'id': 'c554e615-1b70-4dc1-a4aa-aa09ccdacb09',\n",
+      "            'metadata': {'__langserve_endpoint': 'stream_log',\n",
+      "                         '__langserve_version': '0.0.37',\n",
+      "                         '__useragent': 'python-httpx/0.25.2'},\n",
+      "            'name': '<lambda>',\n",
+      "            'start_time': '2024-01-06T03:12:43.099+00:00',\n",
+      "            'streamed_output': [],\n",
+      "            'streamed_output_str': [],\n",
+      "            'tags': ['map:key:agent_scratchpad'],\n",
+      "            'type': 'chain'}})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/<lambda>:3/final_output',\n",
+      "  'value': {'output': 'what does eugene think about cats?'}},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/<lambda>:3/end_time',\n",
+      "  'value': '2024-01-06T03:12:43.100+00:00'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/<lambda>:4/final_output',\n",
+      "  'value': {'output': [AIMessage(content='', additional_kwargs={'function_call': {'name': 'get_eugene_thoughts', 'arguments': '{\\n  \"query\": \"cats\"\\n}'}}),\n",
+      "                       FunctionMessage(content=\"[Document(page_content='cats like fish'), Document(page_content='dogs like sticks')]\", name='get_eugene_thoughts')]}},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/<lambda>:4/end_time',\n",
+      "  'value': '2024-01-06T03:12:43.100+00:00'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/RunnableParallel<input,agent_scratchpad>:2/final_output',\n",
+      "  'value': {'agent_scratchpad': [AIMessage(content='', additional_kwargs={'function_call': {'name': 'get_eugene_thoughts', 'arguments': '{\\n  \"query\": \"cats\"\\n}'}}),\n",
+      "                                 FunctionMessage(content=\"[Document(page_content='cats like fish'), Document(page_content='dogs like sticks')]\", name='get_eugene_thoughts')],\n",
+      "            'input': 'what does eugene think about cats?'}},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/RunnableParallel<input,agent_scratchpad>:2/end_time',\n",
+      "  'value': '2024-01-06T03:12:43.101+00:00'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatPromptTemplate:2',\n",
+      "  'value': {'end_time': None,\n",
+      "            'final_output': None,\n",
+      "            'id': '724c5bf8-d475-45ec-a69a-a4f188dc405f',\n",
+      "            'metadata': {'__langserve_endpoint': 'stream_log',\n",
+      "                         '__langserve_version': '0.0.37',\n",
+      "                         '__useragent': 'python-httpx/0.25.2'},\n",
+      "            'name': 'ChatPromptTemplate',\n",
+      "            'start_time': '2024-01-06T03:12:43.101+00:00',\n",
+      "            'streamed_output': [],\n",
+      "            'streamed_output_str': [],\n",
+      "            'tags': ['seq:step:2'],\n",
+      "            'type': 'prompt'}})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatPromptTemplate:2/final_output',\n",
+      "  'value': {'messages': [SystemMessage(content='You are a helpful assistant.'),\n",
+      "                         HumanMessage(content='what does eugene think about cats?'),\n",
+      "                         AIMessage(content='', additional_kwargs={'function_call': {'name': 'get_eugene_thoughts', 'arguments': '{\\n  \"query\": \"cats\"\\n}'}}),\n",
+      "                         FunctionMessage(content=\"[Document(page_content='cats like fish'), Document(page_content='dogs like sticks')]\", name='get_eugene_thoughts')]}},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/ChatPromptTemplate:2/end_time',\n",
+      "  'value': '2024-01-06T03:12:43.102+00:00'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/LLM:2',\n",
+      "  'value': {'end_time': None,\n",
+      "            'final_output': None,\n",
+      "            'id': '4542ae99-f854-41be-9669-41d8cd7dfb24',\n",
+      "            'metadata': {'__langserve_endpoint': 'stream_log',\n",
+      "                         '__langserve_version': '0.0.37',\n",
+      "                         '__useragent': 'python-httpx/0.25.2'},\n",
+      "            'name': 'LLM',\n",
+      "            'start_time': '2024-01-06T03:12:43.103+00:00',\n",
+      "            'streamed_output': [],\n",
+      "            'streamed_output_str': [],\n",
+      "            'tags': ['seq:step:3'],\n",
+      "            'type': 'llm'}})\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/LLM:2/streamed_output_str/-', 'value': ''},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/LLM:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='')})\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/LLM:2/streamed_output_str/-', 'value': 'E'},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/LLM:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='E')})\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/LLM:2/streamed_output_str/-', 'value': 'ug'},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/LLM:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='ug')})\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/LLM:2/streamed_output_str/-', 'value': 'ene'},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/LLM:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='ene')})\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/LLM:2/streamed_output_str/-', 'value': ' thinks'},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/LLM:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content=' thinks')})\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/LLM:2/streamed_output_str/-', 'value': ' that'},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/LLM:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content=' that')})\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/LLM:2/streamed_output_str/-', 'value': ' cats'},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/LLM:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content=' cats')})\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/LLM:2/streamed_output_str/-', 'value': ' like'},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/LLM:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content=' like')})\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/LLM:2/streamed_output_str/-', 'value': ' fish'},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/LLM:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content=' fish')})\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/LLM:2/streamed_output_str/-', 'value': '.'},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/LLM:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='.')})\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/LLM:2/streamed_output_str/-', 'value': ''},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/LLM:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='')})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/LLM:2/final_output',\n",
+      "  'value': LLMResult(generations=[[ChatGeneration(text='Eugene thinks that cats like fish.', generation_info={'finish_reason': 'stop'}, message=AIMessage(content='Eugene thinks that cats like fish.'))]], llm_output=None, run=None)},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/LLM:2/end_time',\n",
+      "  'value': '2024-01-06T03:12:43.776+00:00'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/OpenAIFunctionsAgentOutputParser:2',\n",
+      "  'value': {'end_time': None,\n",
+      "            'final_output': None,\n",
+      "            'id': '281c8ae8-dbc6-433f-86c8-20b86b3c8eee',\n",
+      "            'metadata': {'__langserve_endpoint': 'stream_log',\n",
+      "                         '__langserve_version': '0.0.37',\n",
+      "                         '__useragent': 'python-httpx/0.25.2'},\n",
+      "            'name': 'OpenAIFunctionsAgentOutputParser',\n",
+      "            'start_time': '2024-01-06T03:12:43.776+00:00',\n",
+      "            'streamed_output': [],\n",
+      "            'streamed_output_str': [],\n",
+      "            'tags': ['seq:step:4'],\n",
+      "            'type': 'parser'}})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/OpenAIFunctionsAgentOutputParser:2/final_output',\n",
+      "  'value': AgentFinish(return_values={'output': 'Eugene thinks that cats like fish.'}, log='Eugene thinks that cats like fish.')},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/OpenAIFunctionsAgentOutputParser:2/end_time',\n",
+      "  'value': '2024-01-06T03:12:43.777+00:00'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/RunnableSequence:2/final_output',\n",
+      "  'value': {'output': AgentFinish(return_values={'output': 'Eugene thinks that cats like fish.'}, log='Eugene thinks that cats like fish.')}},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/RunnableSequence:2/end_time',\n",
+      "  'value': '2024-01-06T03:12:43.778+00:00'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/streamed_output/-',\n",
+      "  'value': {'messages': [AIMessage(content='Eugene thinks that cats like fish.')],\n",
+      "            'output': 'Eugene thinks that cats like fish.'}},\n",
+      " {'op': 'add',\n",
+      "  'path': '/final_output/output',\n",
+      "  'value': 'Eugene thinks that cats like fish.'},\n",
+      " {'op': 'add',\n",
+      "  'path': '/final_output/messages/2',\n",
+      "  'value': AIMessage(content='Eugene thinks that cats like fish.')})\n"
+     ]
+    }
+   ],
+   "source": [
+    "async for chunk in remote_runnable.astream_log({\"input\": \"what does eugene think about cats?\"}):\n",
+    "    print(chunk)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "RunLogPatch({'op': 'replace',\n",
+      "  'path': '',\n",
+      "  'value': {'final_output': None,\n",
+      "            'id': '51c65021-1b40-4a45-81b3-95fc5c5b545a',\n",
+      "            'logs': {},\n",
+      "            'streamed_output': []}})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/LLM',\n",
+      "  'value': {'end_time': None,\n",
+      "            'final_output': None,\n",
+      "            'id': '7a34b2a3-d6c0-4a0b-9939-2bde70a98958',\n",
+      "            'metadata': {'__langserve_endpoint': 'stream_log',\n",
+      "                         '__langserve_version': '0.0.37',\n",
+      "                         '__useragent': 'python-httpx/0.25.2'},\n",
+      "            'name': 'LLM',\n",
+      "            'start_time': '2024-01-06T03:12:43.812+00:00',\n",
+      "            'streamed_output': [],\n",
+      "            'streamed_output_str': [],\n",
+      "            'tags': ['seq:step:3'],\n",
+      "            'type': 'llm'}})\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/LLM/streamed_output_str/-', 'value': ''},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/LLM/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='', additional_kwargs={'function_call': {'name': 'get_eugene_thoughts', 'arguments': ''}})})\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/LLM/streamed_output_str/-', 'value': ''},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/LLM/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='', additional_kwargs={'function_call': {'arguments': '{\\n'}})})\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/LLM/streamed_output_str/-', 'value': ''},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/LLM/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='', additional_kwargs={'function_call': {'arguments': ' '}})})\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/LLM/streamed_output_str/-', 'value': ''},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/LLM/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='', additional_kwargs={'function_call': {'arguments': ' \"'}})})\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/LLM/streamed_output_str/-', 'value': ''},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/LLM/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='', additional_kwargs={'function_call': {'arguments': 'query'}})})\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/LLM/streamed_output_str/-', 'value': ''},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/LLM/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='', additional_kwargs={'function_call': {'arguments': '\":'}})})\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/LLM/streamed_output_str/-', 'value': ''},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/LLM/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='', additional_kwargs={'function_call': {'arguments': ' \"'}})})\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/LLM/streamed_output_str/-', 'value': ''},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/LLM/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='', additional_kwargs={'function_call': {'arguments': 'cats'}})})\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/LLM/streamed_output_str/-', 'value': ''},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/LLM/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='', additional_kwargs={'function_call': {'arguments': '\"\\n'}})})\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/LLM/streamed_output_str/-', 'value': ''},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/LLM/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='', additional_kwargs={'function_call': {'arguments': '}'}})})\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/LLM/streamed_output_str/-', 'value': ''},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/LLM/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='')})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/LLM/final_output',\n",
+      "  'value': LLMResult(generations=[[ChatGeneration(generation_info={'finish_reason': 'function_call'}, message=AIMessage(content='', additional_kwargs={'function_call': {'name': 'get_eugene_thoughts', 'arguments': '{\\n  \"query\": \"cats\"\\n}'}}))]], llm_output=None, run=None)},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/LLM/end_time',\n",
+      "  'value': '2024-01-06T03:12:44.588+00:00'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/streamed_output/-',\n",
+      "  'value': {'actions': [AgentActionMessageLog(tool='get_eugene_thoughts', tool_input={'query': 'cats'}, log=\"\\nInvoking: `get_eugene_thoughts` with `{'query': 'cats'}`\\n\\n\\n\", message_log=[AIMessage(content='', additional_kwargs={'function_call': {'name': 'get_eugene_thoughts', 'arguments': '{\\n  \"query\": \"cats\"\\n}'}})])],\n",
+      "            'messages': [AIMessage(content='', additional_kwargs={'function_call': {'name': 'get_eugene_thoughts', 'arguments': '{\\n  \"query\": \"cats\"\\n}'}})]}},\n",
+      " {'op': 'replace',\n",
+      "  'path': '/final_output',\n",
+      "  'value': {'actions': [AgentActionMessageLog(tool='get_eugene_thoughts', tool_input={'query': 'cats'}, log=\"\\nInvoking: `get_eugene_thoughts` with `{'query': 'cats'}`\\n\\n\\n\", message_log=[AIMessage(content='', additional_kwargs={'function_call': {'name': 'get_eugene_thoughts', 'arguments': '{\\n  \"query\": \"cats\"\\n}'}})])],\n",
+      "            'messages': [AIMessage(content='', additional_kwargs={'function_call': {'name': 'get_eugene_thoughts', 'arguments': '{\\n  \"query\": \"cats\"\\n}'}})]}})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/streamed_output/-',\n",
+      "  'value': {'messages': [FunctionMessage(content=\"[Document(page_content='cats like fish'), Document(page_content='dogs like sticks')]\", name='get_eugene_thoughts')],\n",
+      "            'steps': [{'action': AgentActionMessageLog(tool='get_eugene_thoughts', tool_input={'query': 'cats'}, log=\"\\nInvoking: `get_eugene_thoughts` with `{'query': 'cats'}`\\n\\n\\n\", message_log=[AIMessage(content='', additional_kwargs={'function_call': {'name': 'get_eugene_thoughts', 'arguments': '{\\n  \"query\": \"cats\"\\n}'}})]),\n",
+      "                       'observation': [Document(page_content='cats like fish'),\n",
+      "                                       Document(page_content='dogs like sticks')]}]}},\n",
+      " {'op': 'add',\n",
+      "  'path': '/final_output/steps',\n",
+      "  'value': [{'action': AgentActionMessageLog(tool='get_eugene_thoughts', tool_input={'query': 'cats'}, log=\"\\nInvoking: `get_eugene_thoughts` with `{'query': 'cats'}`\\n\\n\\n\", message_log=[AIMessage(content='', additional_kwargs={'function_call': {'name': 'get_eugene_thoughts', 'arguments': '{\\n  \"query\": \"cats\"\\n}'}})]),\n",
+      "             'observation': [Document(page_content='cats like fish'),\n",
+      "                             Document(page_content='dogs like sticks')]}]},\n",
+      " {'op': 'add',\n",
+      "  'path': '/final_output/messages/1',\n",
+      "  'value': FunctionMessage(content=\"[Document(page_content='cats like fish'), Document(page_content='dogs like sticks')]\", name='get_eugene_thoughts')})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/LLM:2',\n",
+      "  'value': {'end_time': None,\n",
+      "            'final_output': None,\n",
+      "            'id': '8ff3b248-fb7e-4305-bd57-54e21a85bacb',\n",
+      "            'metadata': {'__langserve_endpoint': 'stream_log',\n",
+      "                         '__langserve_version': '0.0.37',\n",
+      "                         '__useragent': 'python-httpx/0.25.2'},\n",
+      "            'name': 'LLM',\n",
+      "            'start_time': '2024-01-06T03:12:44.756+00:00',\n",
+      "            'streamed_output': [],\n",
+      "            'streamed_output_str': [],\n",
+      "            'tags': ['seq:step:3'],\n",
+      "            'type': 'llm'}})\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/LLM:2/streamed_output_str/-', 'value': ''},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/LLM:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='')})\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/LLM:2/streamed_output_str/-', 'value': 'E'},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/LLM:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='E')})\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/LLM:2/streamed_output_str/-', 'value': 'ug'},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/LLM:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='ug')})\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/LLM:2/streamed_output_str/-', 'value': 'ene'},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/LLM:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='ene')})\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/LLM:2/streamed_output_str/-', 'value': ' thinks'},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/LLM:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content=' thinks')})\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/LLM:2/streamed_output_str/-', 'value': ' that'},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/LLM:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content=' that')})\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/LLM:2/streamed_output_str/-', 'value': ' cats'},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/LLM:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content=' cats')})\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/LLM:2/streamed_output_str/-', 'value': ' like'},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/LLM:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content=' like')})\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/LLM:2/streamed_output_str/-', 'value': ' fish'},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/LLM:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content=' fish')})\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/LLM:2/streamed_output_str/-', 'value': '.'},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/LLM:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='.')})\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/LLM:2/streamed_output_str/-', 'value': ''},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/LLM:2/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='')})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/LLM:2/final_output',\n",
+      "  'value': LLMResult(generations=[[ChatGeneration(text='Eugene thinks that cats like fish.', generation_info={'finish_reason': 'stop'}, message=AIMessage(content='Eugene thinks that cats like fish.'))]], llm_output=None, run=None)},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/LLM:2/end_time',\n",
+      "  'value': '2024-01-06T03:12:45.171+00:00'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/streamed_output/-',\n",
+      "  'value': {'messages': [AIMessage(content='Eugene thinks that cats like fish.')],\n",
+      "            'output': 'Eugene thinks that cats like fish.'}},\n",
+      " {'op': 'add',\n",
+      "  'path': '/final_output/output',\n",
+      "  'value': 'Eugene thinks that cats like fish.'},\n",
+      " {'op': 'add',\n",
+      "  'path': '/final_output/messages/2',\n",
+      "  'value': AIMessage(content='Eugene thinks that cats like fish.')})\n"
+     ]
+    }
+   ],
+   "source": [
+    "async for chunk in remote_runnable.astream_log({\"input\": \"what does eugene think about cats?\"}, include_names=[\"LLM\"]):\n",
+    "    print(chunk)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.6"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

examples/configurable_agent_executor/server.py

@@ -0,0 +1,155 @@
+#!/usr/bin/env python
+"""An example that shows how to create a custom agent executor like Runnable.
+
+At the time of writing, there is a bug in the current AgentExecutor that
+prevents it from correctly propagating configuration of the underlying
+runnable. While that bug should be fixed, this is an example shows
+how to create a more complex custom runnable.
+
+Please see documentation for custom agent streaming here:
+
+https://python.langchain.com/docs/modules/agents/how_to/streaming#stream-tokens
+
+**ATTENTION**
+To support streaming individual tokens you will need to manually set the streaming=True
+on the LLM and use the stream_log endpoint rather than stream endpoint.
+"""
+from typing import Any, AsyncIterator, Dict, List, Optional, cast
+
+from fastapi import FastAPI
+from langchain.agents import AgentExecutor, tool
+from langchain.agents.format_scratchpad import format_to_openai_functions
+from langchain.agents.output_parsers import OpenAIFunctionsAgentOutputParser
+from langchain.chat_models import ChatOpenAI
+from langchain.embeddings import OpenAIEmbeddings
+from langchain.prompts import ChatPromptTemplate, MessagesPlaceholder
+from langchain.pydantic_v1 import BaseModel
+from langchain.tools.render import format_tool_to_openai_function
+from langchain.vectorstores import FAISS
+from langchain_core.runnables import (
+    ConfigurableField,
+    ConfigurableFieldSpec,
+    Runnable,
+    RunnableConfig,
+)
+from langchain_core.runnables.utils import Input, Output
+
+from langserve import add_routes
+
+vectorstore = FAISS.from_texts(
+    ["cats like fish", "dogs like sticks"], embedding=OpenAIEmbeddings()
+)
+retriever = vectorstore.as_retriever()
+
+
+@tool
+def get_eugene_thoughts(query: str) -> list:
+    """Returns Eugene's thoughts on a topic."""
+    return retriever.get_relevant_documents(query)
+
+
+tools = [get_eugene_thoughts]
+
+prompt = ChatPromptTemplate.from_messages(
+    [
+        ("system", "You are a helpful assistant."),
+        ("user", "{input}"),
+        MessagesPlaceholder(variable_name="agent_scratchpad"),
+    ]
+)
+
+# We need to set streaming=True on the LLM to support streaming individual tokens.
+# when using the stream_log endpoint.
+# .stream for agents streams action observation pairs not individual tokens.
+llm = ChatOpenAI(temperature=0, streaming=True).configurable_fields(
+    temperature=ConfigurableField(
+        id="llm_temperature",
+        name="LLM Temperature",
+        description="The temperature of the LLM",
+    )
+)
+
+llm_with_tools = llm.bind(
+    functions=[format_tool_to_openai_function(t) for t in tools]
+).with_config({"run_name": "LLM"})
+
+
+class CustomAgentExecutor(Runnable):
+    """A custom runnable that will be used by the agent executor."""
+
+    def __init__(self, **kwargs):
+        """Initialize the runnable."""
+        super().__init__(**kwargs)
+        self.agent = (
+            {
+                "input": lambda x: x["input"],
+                "agent_scratchpad": lambda x: format_to_openai_functions(
+                    x["intermediate_steps"]
+                ),
+            }
+            | prompt
+            | llm_with_tools
+            | OpenAIFunctionsAgentOutputParser()
+        )
+
+    def invoke(self, input: Input, config: Optional[RunnableConfig] = None) -> Output:
+        """Will not be used."""
+        raise NotImplementedError()
+
+    @property
+    def config_specs(self) -> List[ConfigurableFieldSpec]:
+        return self.agent.config_specs
+
+    async def astream(
+        self,
+        input: Input,
+        config: Optional[RunnableConfig] = None,
+        **kwargs: Optional[Any],
+    ) -> AsyncIterator[Output]:
+        """Stream the agent's output."""
+        configurable = cast(Dict[str, Any], config.pop("configurable", {}))
+
+        if configurable:
+            configured_agent = self.agent.with_config(
+                {
+                    "configurable": configurable,
+                }
+            )
+        else:
+            configured_agent = self.agent
+
+        executor = AgentExecutor(
+            agent=configured_agent,
+            tools=tools,
+        ).with_config({"run_name": "executor"})
+
+        async for output in executor.astream(input, config=config, **kwargs):
+            yield output
+
+
+app = FastAPI()
+
+
+# We need to add these input/output schemas because the current AgentExecutor
+# is lacking in schemas.
+class Input(BaseModel):
+    input: str
+
+
+class Output(BaseModel):
+    output: Any
+
+
+runnable = CustomAgentExecutor()
+
+# Add routes to the app for using the custom agent executor.
+add_routes(
+    app,
+    runnable.with_types(input_type=Input, output_type=Output),
+    disabled_endpoints=["invoke", "batch"],  # not implemented
+)
+
+if __name__ == "__main__":
+    import uvicorn
+
+    uvicorn.run(app, host="localhost", port=8000)

examples/configurable_chain/server.py

@@ -59,13 +59,32 @@ prompt = PromptTemplate.from_template(
 )
 chain = prompt | model | StrOutputParser()
 
-# Add routes requires you to specify which config keys are accepted
-# specifically, you must accept `configurable` as a config key.
-add_routes(app, chain, path="/configurable_temp", config_keys=["configurable"])
+add_routes(app, chain, path="/configurable_temp")
 
 
 ###############################################################################
-#             EXAMPLE 2: Configure fields based on Request metadata           #
+#                EXAMPLE 2: Configure prompt based on RunnableConfig          #
+###############################################################################
+configurable_prompt = PromptTemplate.from_template(
+    "tell me a joke about {topic}."
+).configurable_alternatives(
+    ConfigurableField(
+        id="prompt",
+        name="Prompt",
+        description="The prompt to use. Must contain {topic}.",
+    ),
+    default_key="joke",
+    fact=PromptTemplate.from_template(
+        "tell me a fact about {topic} in {language} language."
+    ),
+)
+prompt_chain = configurable_prompt | model | StrOutputParser()
+
+add_routes(app, prompt_chain, path="/configurable_prompt")
+
+
+###############################################################################
+#             EXAMPLE 3: Configure fields based on Request metadata           #
 ###############################################################################
 
 
@@ -96,7 +115,6 @@ add_routes(
     app,
     dynamic_auth_chain,
     path="/auth_from_header",
-    config_keys=["configurable"],
     per_req_config_modifier=fetch_api_key_from_header,
 )
 

examples/configurable_retrieval/client.ipynb

@@ -0,0 +1,168 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Client\n",
+    "\n",
+    "Demo of a client interacting with a configurable retriever (see server code)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "You can interact with this via API directly"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'output': [{'page_content': 'cats like fish',\n",
+       "   'metadata': {},\n",
+       "   'type': 'Document'},\n",
+       "  {'page_content': 'dogs like sticks', 'metadata': {}, 'type': 'Document'}],\n",
+       " 'callback_events': [],\n",
+       " 'metadata': {'run_id': 'f375cdf6-2848-4976-9565-f69e175c24ce'}}"
+      ]
+     },
+     "execution_count": 1,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "import requests\n",
+    "\n",
+    "inputs = {\"input\": \"cat\"}\n",
+    "response = requests.post(\"http://localhost:8000/invoke\", json=inputs)\n",
+    "\n",
+    "response.json()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "You can also interact with this via the RemoteRunnable interface (to use in other chains)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "from langserve import RemoteRunnable\n",
+    "\n",
+    "remote_runnable = RemoteRunnable(\"http://localhost:8000/\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Remote runnable has the same interface as local runnables"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[Document(page_content='cats like fish'),\n",
+       " Document(page_content='dogs like sticks')]"
+      ]
+     },
+     "execution_count": 3,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "await remote_runnable.ainvoke(\"cat\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[Document(page_content='cats like fish'),\n",
+       " Document(page_content='dogs like sticks')]"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "await remote_runnable.ainvoke(\"cat\", {\"configurable\": {\"collection_name\": \"Index 1\"}})"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[Document(page_content='x_n+1=a * xn * (1-xn)')]"
+      ]
+     },
+     "execution_count": 5,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "await remote_runnable.ainvoke(\"cat\", {\"configurable\": {\"collection_name\": \"Index 2\"}})"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.6"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

examples/configurable_retrieval/server.py

@@ -0,0 +1,126 @@
+#!/usr/bin/env python
+"""A more complex example that shows how to configure index name at run time."""
+from typing import Any, Iterable, List, Optional, Type
+
+from fastapi import FastAPI
+from langchain.embeddings import OpenAIEmbeddings
+from langchain.schema import Document
+from langchain.schema.embeddings import Embeddings
+from langchain.schema.retriever import BaseRetriever
+from langchain.schema.runnable import (
+    ConfigurableFieldSingleOption,
+    RunnableConfig,
+    RunnableSerializable,
+)
+from langchain.schema.vectorstore import VST
+from langchain.vectorstores import FAISS, VectorStore
+
+from langserve import add_routes
+from langserve.pydantic_v1 import BaseModel, Field
+
+vectorstore1 = FAISS.from_texts(
+    ["cats like fish", "dogs like sticks"], embedding=OpenAIEmbeddings()
+)
+
+vectorstore2 = FAISS.from_texts(["x_n+1=a * xn * (1-xn)"], embedding=OpenAIEmbeddings())
+
+
+app = FastAPI(
+    title="LangChain Server",
+    version="1.0",
+    description="Spin up a simple api server using Langchain's Runnable interfaces",
+)
+
+
+class UnderlyingVectorStore(VectorStore):
+    """This is a fake vectorstore for demo purposes."""
+
+    def __init__(self, collection_name: str) -> None:
+        """Fake vectorstore that has a collection name."""
+        self.collection_name = collection_name
+
+    def as_retriever(self) -> BaseRetriever:
+        if self.collection_name == "index1":
+            return vectorstore1.as_retriever()
+        elif self.collection_name == "index2":
+            return vectorstore2.as_retriever()
+        else:
+            raise NotImplementedError(
+                f"No retriever for collection {self.collection_name}"
+            )
+
+    def add_texts(
+        self,
+        texts: Iterable[str],
+        metadatas: Optional[List[dict]] = None,
+        **kwargs: Any,
+    ) -> List[str]:
+        raise NotImplementedError()
+
+    @classmethod
+    def from_texts(
+        cls: Type[VST],
+        texts: List[str],
+        embedding: Embeddings,
+        metadatas: Optional[List[dict]] = None,
+        **kwargs: Any,
+    ) -> VST:
+        raise NotImplementedError()
+
+    def similarity_search(
+        self, embedding: List[float], k: int = 4, **kwargs: Any
+    ) -> List[Document]:
+        raise NotImplementedError()
+
+
+class ConfigurableRetriever(RunnableSerializable[str, List[Document]]):
+    """Create a custom retriever that can be configured by the user.
+
+    This is an example of how to create a custom runnable that can be configured
+    to use a different collection name at run time.
+
+    Configuration involves instantiating a VectorStore with a collection name.
+    at run time, so the underlying vectorstore should be *cheap* to instantiate.
+
+    For example, it should not be making any network requests at instantiation time.
+
+    Make sure that the vectorstore you use meets this criteria.
+    """
+
+    collection_name: str
+
+    def invoke(
+        self, input: str, config: Optional[RunnableConfig] = None
+    ) -> List[Document]:
+        """Invoke the retriever."""
+        vectorstore = UnderlyingVectorStore(self.collection_name)
+        retriever = vectorstore.as_retriever()
+        return retriever.invoke(input, config=config)
+
+
+configurable_collection_name = ConfigurableRetriever(
+    collection_name="index1"
+).configurable_fields(
+    collection_name=ConfigurableFieldSingleOption(
+        id="collection_name",
+        name="Collection Name",
+        description="The name of the collection to use for the retriever.",
+        options={
+            "Index 1": "index1",
+            "Index 2": "index2",
+        },
+        default="Index 1",
+    )
+)
+
+
+class Request(BaseModel):
+    __root__: str = Field(default="cat", description="Search query")
+
+
+add_routes(app, configurable_collection_name.with_types(input_type=Request))
+
+if __name__ == "__main__":
+    import uvicorn
+
+    uvicorn.run(app, host="localhost", port=8000)

examples/conversational_retrieval_chain/client.ipynb

@@ -18,7 +18,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 1,
+   "execution_count": 11,
    "metadata": {
     "tags": []
    },
@@ -26,10 +26,15 @@
     {
      "data": {
       "text/plain": [
-       "{'output': {'answer': 'Cats like fish.'}, 'callback_events': []}"
+       "{'output': {'content': 'Based on the given context, the information we have about Harrison is that he worked at Kensho.',\n",
+       "  'additional_kwargs': {},\n",
+       "  'type': 'ai',\n",
+       "  'example': False},\n",
+       " 'callback_events': [],\n",
+       " 'metadata': {'run_id': '3455df2b-93f8-4e67-b1a3-27f90670cf7b'}}"
       ]
      },
-     "execution_count": 1,
+     "execution_count": 11,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -37,7 +42,7 @@
    "source": [
     "import requests\n",
     "\n",
-    "inputs = {\"input\": {\"question\": \"what do cats like?\", \"chat_history\": []}}\n",
+    "inputs = {\"input\": {\"question\": \"what do you know about harrison\", \"chat_history\": []}}\n",
     "response = requests.post(\"http://localhost:8000/invoke\", json=inputs)\n",
     "\n",
     "response.json()"
@@ -52,7 +57,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
+   "execution_count": 12,
    "metadata": {
     "tags": []
    },
@@ -72,7 +77,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": 13,
    "metadata": {
     "tags": []
    },
@@ -80,21 +85,21 @@
     {
      "data": {
       "text/plain": [
-       "{'answer': 'Cats like fish.'}"
+       "AIMessage(content='Based on the given context, the only information we have about Harrison is that he worked at Kensho.')"
       ]
      },
-     "execution_count": 3,
+     "execution_count": 13,
      "metadata": {},
      "output_type": "execute_result"
     }
    ],
    "source": [
-    "await remote_runnable.ainvoke({\"question\": \"what do cats like?\", \"chat_history\": []})"
+    "await remote_runnable.ainvoke({\"question\": \"what do you know about harrison\", \"chat_history\": []})"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
+   "execution_count": 14,
    "metadata": {
     "tags": []
    },
@@ -102,26 +107,501 @@
     {
      "data": {
       "text/plain": [
-       "{'answer': 'Cats like fish.'}"
+       "AIMessage(content='Harrison worked at Kensho.')"
       ]
      },
-     "execution_count": 6,
+     "execution_count": 14,
      "metadata": {},
      "output_type": "execute_result"
     }
    ],
    "source": [
     "await remote_runnable.ainvoke(\n",
-    "    {\"question\": \"what do cats like?\", \"chat_history\": [(\"hi\", \"hi\")]}\n",
+    "    {\"question\": \"what do you know about harrison\", \"chat_history\": [(\"hi\", \"hi\")]}\n",
     ")"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 16,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "content=''\n",
+      "content='H'\n",
+      "content='arrison'\n",
+      "content=' worked'\n",
+      "content=' at'\n",
+      "content=' Kens'\n",
+      "content='ho'\n",
+      "content='.'\n",
+      "content=''\n"
+     ]
+    }
+   ],
+   "source": [
+    "async for chunk in remote_runnable.astream(\n",
+    "    {\"question\": \"what do you know about harrison\", \"chat_history\": [(\"hi\", \"hi\")]}\n",
+    "):\n",
+    "    print(chunk)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
    "metadata": {},
-   "outputs": [],
-   "source": []
+   "source": [
+    "stream log shows all intermediate steps as well!"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 18,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "RunLogPatch({'op': 'replace',\n",
+      "  'path': '',\n",
+      "  'value': {'final_output': None,\n",
+      "            'id': '2ff5a98d-49f0-40ae-92fe-489c3047d1c3',\n",
+      "            'logs': {},\n",
+      "            'streamed_output': []}})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/RunnableParallel',\n",
+      "  'value': {'end_time': None,\n",
+      "            'final_output': None,\n",
+      "            'id': 'ffdda3c9-a0ba-49a6-af18-c01748c31801',\n",
+      "            'metadata': {},\n",
+      "            'name': 'RunnableParallel',\n",
+      "            'start_time': '2023-11-16T15:59:23.348',\n",
+      "            'streamed_output_str': [],\n",
+      "            'tags': ['seq:step:1'],\n",
+      "            'type': 'chain'}})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/RunnableSequence',\n",
+      "  'value': {'end_time': None,\n",
+      "            'final_output': None,\n",
+      "            'id': 'a9da3f2c-f3ae-44c1-a2f4-1035faa0d1c2',\n",
+      "            'metadata': {},\n",
+      "            'name': 'RunnableSequence',\n",
+      "            'start_time': '2023-11-16T15:59:23.349',\n",
+      "            'streamed_output_str': [],\n",
+      "            'tags': ['map:key:standalone_question'],\n",
+      "            'type': 'chain'}})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/RunnableParallel:2',\n",
+      "  'value': {'end_time': None,\n",
+      "            'final_output': None,\n",
+      "            'id': '4d32b08f-a989-4c01-8068-479bed506d99',\n",
+      "            'metadata': {},\n",
+      "            'name': 'RunnableParallel',\n",
+      "            'start_time': '2023-11-16T15:59:23.349',\n",
+      "            'streamed_output_str': [],\n",
+      "            'tags': ['seq:step:1'],\n",
+      "            'type': 'chain'}})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/<lambda>',\n",
+      "  'value': {'end_time': None,\n",
+      "            'final_output': None,\n",
+      "            'id': 'c7b9bd78-cbb4-41f9-b31a-50a5f8940a8d',\n",
+      "            'metadata': {},\n",
+      "            'name': '<lambda>',\n",
+      "            'start_time': '2023-11-16T15:59:23.350',\n",
+      "            'streamed_output_str': [],\n",
+      "            'tags': ['map:key:chat_history'],\n",
+      "            'type': 'chain'}})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/<lambda>/final_output',\n",
+      "  'value': {'output': '\\nHuman: hi\\nAssistant: hi'}},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/<lambda>/end_time',\n",
+      "  'value': '2023-11-16T15:59:23.350'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/RunnableParallel:2/final_output',\n",
+      "  'value': {'chat_history': '\\nHuman: hi\\nAssistant: hi'}},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/RunnableParallel:2/end_time',\n",
+      "  'value': '2023-11-16T15:59:23.351'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/PromptTemplate',\n",
+      "  'value': {'end_time': None,\n",
+      "            'final_output': None,\n",
+      "            'id': 'ef9f8729-2d4a-4887-86a8-a284d64f5882',\n",
+      "            'metadata': {},\n",
+      "            'name': 'PromptTemplate',\n",
+      "            'start_time': '2023-11-16T15:59:23.351',\n",
+      "            'streamed_output_str': [],\n",
+      "            'tags': ['seq:step:2'],\n",
+      "            'type': 'prompt'}})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/PromptTemplate/final_output',\n",
+      "  'value': StringPromptValue(text='Given the following conversation and a follow up question, rephrase the \\nfollow up question to be a standalone question, in its original language.\\n\\nChat History:\\n\\nHuman: hi\\nAssistant: hi\\nFollow Up Input: what do you know about harrison\\nStandalone question:')},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/PromptTemplate/end_time',\n",
+      "  'value': '2023-11-16T15:59:23.351'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI',\n",
+      "  'value': {'end_time': None,\n",
+      "            'final_output': None,\n",
+      "            'id': '7cda5923-ed2a-42ea-aee7-a2391371ff2f',\n",
+      "            'metadata': {},\n",
+      "            'name': 'ChatOpenAI',\n",
+      "            'start_time': '2023-11-16T15:59:23.352',\n",
+      "            'streamed_output_str': [],\n",
+      "            'tags': ['seq:step:3'],\n",
+      "            'type': 'llm'}})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/StrOutputParser',\n",
+      "  'value': {'end_time': None,\n",
+      "            'final_output': None,\n",
+      "            'id': '6a0e199d-51fa-4306-8a7d-054444c6855f',\n",
+      "            'metadata': {},\n",
+      "            'name': 'StrOutputParser',\n",
+      "            'start_time': '2023-11-16T15:59:24.613',\n",
+      "            'streamed_output_str': [],\n",
+      "            'tags': ['seq:step:4'],\n",
+      "            'type': 'parser'}})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/RunnableParallel:3',\n",
+      "  'value': {'end_time': None,\n",
+      "            'final_output': None,\n",
+      "            'id': '74a13de1-c2d1-43c0-813a-7ce0246118b3',\n",
+      "            'metadata': {},\n",
+      "            'name': 'RunnableParallel',\n",
+      "            'start_time': '2023-11-16T15:59:24.616',\n",
+      "            'streamed_output_str': [],\n",
+      "            'tags': ['seq:step:2'],\n",
+      "            'type': 'chain'}})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/RunnableSequence:2',\n",
+      "  'value': {'end_time': None,\n",
+      "            'final_output': None,\n",
+      "            'id': '625e2183-7ee5-4f1f-8f47-56541afc96ee',\n",
+      "            'metadata': {},\n",
+      "            'name': 'RunnableSequence',\n",
+      "            'start_time': '2023-11-16T15:59:24.619',\n",
+      "            'streamed_output_str': [],\n",
+      "            'tags': ['map:key:context'],\n",
+      "            'type': 'chain'}})\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/ChatOpenAI/streamed_output_str/-', 'value': ''})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI/streamed_output_str/-',\n",
+      "  'value': 'What'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI/streamed_output_str/-',\n",
+      "  'value': ' information'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI/streamed_output_str/-',\n",
+      "  'value': ' do'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI/streamed_output_str/-',\n",
+      "  'value': ' you'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI/streamed_output_str/-',\n",
+      "  'value': ' have'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI/streamed_output_str/-',\n",
+      "  'value': ' about'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI/streamed_output_str/-',\n",
+      "  'value': ' Harrison'})\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/ChatOpenAI/streamed_output_str/-', 'value': '?'})\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/ChatOpenAI/streamed_output_str/-', 'value': ''})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI/final_output',\n",
+      "  'value': LLMResult(generations=[[ChatGenerationChunk(text='What information do you have about Harrison?', generation_info={'finish_reason': 'stop'}, message=AIMessageChunk(content='What information do you have about Harrison?'))]], llm_output=None, run=None)},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI/end_time',\n",
+      "  'value': '2023-11-16T15:59:24.832'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/StrOutputParser/final_output',\n",
+      "  'value': {'output': 'What information do you have about Harrison?'}},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/StrOutputParser/end_time',\n",
+      "  'value': '2023-11-16T15:59:24.833'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/RunnableSequence/final_output',\n",
+      "  'value': {'output': 'What information do you have about Harrison?'}},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/RunnableSequence/end_time',\n",
+      "  'value': '2023-11-16T15:59:24.834'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/RunnableParallel/final_output',\n",
+      "  'value': {'standalone_question': 'What information do you have about '\n",
+      "                                   'Harrison?'}},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/RunnableParallel/end_time',\n",
+      "  'value': '2023-11-16T15:59:24.835'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/RunnableLambda',\n",
+      "  'value': {'end_time': None,\n",
+      "            'final_output': None,\n",
+      "            'id': '71b2e8dc-753f-4bf2-83de-b87b26828370',\n",
+      "            'metadata': {},\n",
+      "            'name': 'RunnableLambda',\n",
+      "            'start_time': '2023-11-16T15:59:24.837',\n",
+      "            'streamed_output_str': [],\n",
+      "            'tags': ['seq:step:1'],\n",
+      "            'type': 'chain'}})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/RunnableLambda/final_output',\n",
+      "  'value': {'output': 'What information do you have about Harrison?'}},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/RunnableLambda/end_time',\n",
+      "  'value': '2023-11-16T15:59:24.837'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/Retriever',\n",
+      "  'value': {'end_time': None,\n",
+      "            'final_output': None,\n",
+      "            'id': 'd3d6254d-d073-478f-b00b-bc27eafa24fd',\n",
+      "            'metadata': {},\n",
+      "            'name': 'Retriever',\n",
+      "            'start_time': '2023-11-16T15:59:24.839',\n",
+      "            'streamed_output_str': [],\n",
+      "            'tags': ['seq:step:2', 'FAISS', 'OpenAIEmbeddings'],\n",
+      "            'type': 'retriever'}})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/<lambda>:2',\n",
+      "  'value': {'end_time': None,\n",
+      "            'final_output': None,\n",
+      "            'id': '77f7132a-d98f-4121-89cf-10e462c26496',\n",
+      "            'metadata': {},\n",
+      "            'name': '<lambda>',\n",
+      "            'start_time': '2023-11-16T15:59:24.839',\n",
+      "            'streamed_output_str': [],\n",
+      "            'tags': ['map:key:question'],\n",
+      "            'type': 'chain'}})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/<lambda>:2/final_output',\n",
+      "  'value': {'output': 'What information do you have about Harrison?'}},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/<lambda>:2/end_time',\n",
+      "  'value': '2023-11-16T15:59:24.840'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/Retriever/final_output',\n",
+      "  'value': {'documents': [Document(page_content='harrison worked at kensho')]}},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/Retriever/end_time',\n",
+      "  'value': '2023-11-16T15:59:25.074'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/_combine_documents',\n",
+      "  'value': {'end_time': None,\n",
+      "            'final_output': None,\n",
+      "            'id': '82aecf3e-ca9d-4b48-a281-03f8e834ea62',\n",
+      "            'metadata': {},\n",
+      "            'name': '_combine_documents',\n",
+      "            'start_time': '2023-11-16T15:59:25.075',\n",
+      "            'streamed_output_str': [],\n",
+      "            'tags': ['seq:step:3'],\n",
+      "            'type': 'chain'}})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/_combine_documents/final_output',\n",
+      "  'value': {'output': 'harrison worked at kensho'}},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/_combine_documents/end_time',\n",
+      "  'value': '2023-11-16T15:59:25.075'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/RunnableSequence:2/final_output',\n",
+      "  'value': {'output': 'harrison worked at kensho'}},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/RunnableSequence:2/end_time',\n",
+      "  'value': '2023-11-16T15:59:25.075'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/RunnableParallel:3/final_output',\n",
+      "  'value': {'context': 'harrison worked at kensho',\n",
+      "            'question': 'What information do you have about Harrison?'}},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/RunnableParallel:3/end_time',\n",
+      "  'value': '2023-11-16T15:59:25.076'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatPromptTemplate',\n",
+      "  'value': {'end_time': None,\n",
+      "            'final_output': None,\n",
+      "            'id': 'b37f36ce-3a27-4402-81c4-6893f03ec179',\n",
+      "            'metadata': {},\n",
+      "            'name': 'ChatPromptTemplate',\n",
+      "            'start_time': '2023-11-16T15:59:25.076',\n",
+      "            'streamed_output_str': [],\n",
+      "            'tags': ['seq:step:3'],\n",
+      "            'type': 'prompt'}})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatPromptTemplate/final_output',\n",
+      "  'value': {'messages': [HumanMessage(content='Answer the question based only on the following context:\\nharrison worked at kensho\\n\\nQuestion: What information do you have about Harrison?\\n')]}},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/ChatPromptTemplate/end_time',\n",
+      "  'value': '2023-11-16T15:59:25.077'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2',\n",
+      "  'value': {'end_time': None,\n",
+      "            'final_output': None,\n",
+      "            'id': 'b1179088-eb7c-49c5-af1e-bd09c48408bf',\n",
+      "            'metadata': {},\n",
+      "            'name': 'ChatOpenAI',\n",
+      "            'start_time': '2023-11-16T15:59:25.078',\n",
+      "            'streamed_output_str': [],\n",
+      "            'tags': ['seq:step:4'],\n",
+      "            'type': 'llm'}})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='')})\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/ChatOpenAI:2/streamed_output_str/-', 'value': ''})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='Based')})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output_str/-',\n",
+      "  'value': 'Based'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content=' on')})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output_str/-',\n",
+      "  'value': ' on'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content=' the')})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output_str/-',\n",
+      "  'value': ' the'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content=' given')})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output_str/-',\n",
+      "  'value': ' given'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content=' context')})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output_str/-',\n",
+      "  'value': ' context'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content=',')})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output_str/-',\n",
+      "  'value': ','})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content=' the')})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output_str/-',\n",
+      "  'value': ' the'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content=' only')})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output_str/-',\n",
+      "  'value': ' only'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content=' information')})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output_str/-',\n",
+      "  'value': ' information'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content=' we')})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output_str/-',\n",
+      "  'value': ' we'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content=' have')})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output_str/-',\n",
+      "  'value': ' have'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content=' about')})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output_str/-',\n",
+      "  'value': ' about'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content=' Harrison')})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output_str/-',\n",
+      "  'value': ' Harrison'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content=' is')})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output_str/-',\n",
+      "  'value': ' is'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content=' that')})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output_str/-',\n",
+      "  'value': ' that'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content=' he')})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output_str/-',\n",
+      "  'value': ' he'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content=' worked')})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output_str/-',\n",
+      "  'value': ' worked'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content=' at')})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output_str/-',\n",
+      "  'value': ' at'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content=' Kens')})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output_str/-',\n",
+      "  'value': ' Kens'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='ho')})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output_str/-',\n",
+      "  'value': 'ho'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='.')})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/streamed_output_str/-',\n",
+      "  'value': '.'})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/streamed_output/-',\n",
+      "  'value': AIMessageChunk(content='')})\n",
+      "RunLogPatch({'op': 'add', 'path': '/logs/ChatOpenAI:2/streamed_output_str/-', 'value': ''})\n",
+      "RunLogPatch({'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/final_output',\n",
+      "  'value': LLMResult(generations=[[ChatGenerationChunk(text='Based on the given context, the only information we have about Harrison is that he worked at Kensho.', generation_info={'finish_reason': 'stop'}, message=AIMessageChunk(content='Based on the given context, the only information we have about Harrison is that he worked at Kensho.'))]], llm_output=None, run=None)},\n",
+      " {'op': 'add',\n",
+      "  'path': '/logs/ChatOpenAI:2/end_time',\n",
+      "  'value': '2023-11-16T15:59:26.547'})\n",
+      "RunLogPatch({'op': 'replace',\n",
+      "  'path': '/final_output',\n",
+      "  'value': {'output': AIMessageChunk(content='Based on the given context, the only information we have about Harrison is that he worked at Kensho.')}})\n"
+     ]
+    }
+   ],
+   "source": [
+    "async for chunk in remote_runnable.astream_log(\n",
+    "    {\"question\": \"what do you know about harrison\", \"chat_history\": [(\"hi\", \"hi\")]}\n",
+    "):\n",
+    "    print(chunk)"
+   ]
   }
  ],
  "metadata": {

examples/conversational_retrieval_chain/server.py

@@ -1,24 +1,84 @@
 #!/usr/bin/env python
-"""Example LangChain server exposes a conversational retrieval chain."""
+"""Example LangChain server exposes a conversational retrieval chain.
+
+Follow the reference here:
+
+https://python.langchain.com/docs/expression_language/cookbook/retrieval#conversational-retrieval-chain
+
+To run this example, you will need to install the following packages:
+pip install langchain openai faiss-cpu tiktoken
+"""  # noqa: F401
+
+from operator import itemgetter
 from typing import List, Tuple
 
 from fastapi import FastAPI
-from langchain.chains import ConversationalRetrievalChain
 from langchain.chat_models import ChatOpenAI
 from langchain.embeddings import OpenAIEmbeddings
-from langchain.pydantic_v1 import BaseModel, Field
+from langchain.prompts import ChatPromptTemplate
+from langchain.prompts.prompt import PromptTemplate
+from langchain.schema import format_document
+from langchain.schema.output_parser import StrOutputParser
+from langchain.schema.runnable import RunnableMap, RunnablePassthrough
 from langchain.vectorstores import FAISS
 
 from langserve import add_routes
+from langserve.pydantic_v1 import BaseModel, Field
+
+_TEMPLATE = """Given the following conversation and a follow up question, rephrase the 
+follow up question to be a standalone question, in its original language.
+
+Chat History:
+{chat_history}
+Follow Up Input: {question}
+Standalone question:"""
+CONDENSE_QUESTION_PROMPT = PromptTemplate.from_template(_TEMPLATE)
+
+ANSWER_TEMPLATE = """Answer the question based only on the following context:
+{context}
+
+Question: {question}
+"""
+ANSWER_PROMPT = ChatPromptTemplate.from_template(ANSWER_TEMPLATE)
+
+DEFAULT_DOCUMENT_PROMPT = PromptTemplate.from_template(template="{page_content}")
+
+
+def _combine_documents(
+    docs, document_prompt=DEFAULT_DOCUMENT_PROMPT, document_separator="\n\n"
+):
+    """Combine documents into a single string."""
+    doc_strings = [format_document(doc, document_prompt) for doc in docs]
+    return document_separator.join(doc_strings)
+
+
+def _format_chat_history(chat_history: List[Tuple]) -> str:
+    """Format chat history into a string."""
+    buffer = ""
+    for dialogue_turn in chat_history:
+        human = "Human: " + dialogue_turn[0]
+        ai = "Assistant: " + dialogue_turn[1]
+        buffer += "\n" + "\n".join([human, ai])
+    return buffer
+
 
 vectorstore = FAISS.from_texts(
-    ["cats like fish", "dogs like sticks"], embedding=OpenAIEmbeddings()
+    ["harrison worked at kensho"], embedding=OpenAIEmbeddings()
 )
 retriever = vectorstore.as_retriever()
 
-model = ChatOpenAI()
-
-chain = ConversationalRetrievalChain.from_llm(model, retriever)
+_inputs = RunnableMap(
+    standalone_question=RunnablePassthrough.assign(
+        chat_history=lambda x: _format_chat_history(x["chat_history"])
+    )
+    | CONDENSE_QUESTION_PROMPT
+    | ChatOpenAI(temperature=0)
+    | StrOutputParser(),
+)
+_context = {
+    "context": itemgetter("standalone_question") | retriever | _combine_documents,
+    "question": lambda x: x["standalone_question"],
+}
 
 
 # User input
@@ -32,9 +92,10 @@ class ChatHistory(BaseModel):
     question: str
 
 
-chain = ConversationalRetrievalChain.from_llm(model, retriever).with_types(
-    input_type=ChatHistory
+conversational_qa_chain = (
+    _inputs | _context | ANSWER_PROMPT | ChatOpenAI() | StrOutputParser()
 )
+chain = conversational_qa_chain.with_types(input_type=ChatHistory)
 
 app = FastAPI(
     title="LangChain Server",
@@ -45,7 +106,7 @@ app = FastAPI(
 # /invoke
 # /batch
 # /stream
-add_routes(app, chain)
+add_routes(app, chain, enable_feedback_endpoint=True)
 
 if __name__ == "__main__":
     import uvicorn

examples/local_llm/client.ipynb

@@ -0,0 +1,301 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Local LLM\n",
+    "\n",
+    "Here, we'll use a server that's serving a local LLM.\n",
+    "\n",
+    "**Attention** This is OK for prototyping / dev usage, but should not be used for production cases when there might be concurrent requests from different users. As of the time of writing, Ollama is designed for single user and cannot handle concurrent requests see this issue: https://github.com/ollama/ollama/issues/358"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "from langchain.prompts.chat import ChatPromptTemplate"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 15,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "from langserve import RemoteRunnable\n",
+    "\n",
+    "model = RemoteRunnable(\"http://localhost:8000/ollama/\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Let's test out the standard interface of a chat model."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "prompt = \"Tell me a 3 sentence story about a cat.\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 17,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "AIMessage(content=\"\\nSure! Here is a three sentence story about a cat:\\n\\nMittens the cat purred contentedly on the windowsill, basking in the warm sunlight. Suddenly, a bird perched nearby and Mittens' ears perked up, ready to pounce. With lightning quick reflexes, Mittens leapt into the air, but the bird had flown away, leaving Mittens to settle for just lounging in the sun once again.\")"
+      ]
+     },
+     "execution_count": 17,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "model.invoke(prompt)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 18,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "AIMessage(content=\"\\nSure! Here is a three sentence story about a cat:\\n\\nMittens the cat purred contentedly on the windowsill, basking in the warm sunlight. Suddenly, a bird flew by, catching Mittens' attention and causing her to leap into action. With lightning quick reflexes, Mittens pounced on the bird, but it flew away just in time, leaving Mittens frustrated but still purring happily.\")"
+      ]
+     },
+     "execution_count": 18,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "await model.ainvoke(prompt)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Batched API works, but b/c ollama does not support parallelism, it's no faster than using .invoke twice."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 21,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "CPU times: user 7.65 ms, sys: 6.57 ms, total: 14.2 ms\n",
+      "Wall time: 5.51 s\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "[AIMessage(content='\\nSure! Here is a three sentence story about a cat:\\n\\nMr. Whiskers was a sleek black cat with bright green eyes. He spent his days lounging in the sunbeams that streamed through the living room window, chasing the occasional fly, and purring contentedly. Despite his lazy demeanor, Mr. Whiskers was always on the lookout for a warm lap to curl up in.'),\n",
+       " AIMessage(content='\\nSure! Here is a three sentence story about a cat:\\n\\nMittens the cat purred contentedly on the windowsill, basking in the warm sunlight that streamed through the glass. Suddenly, a tiny bird perched on the ledge outside, tweeting nervously as it eyed the cat with suspicion. Without hesitation, Mittens pounced, snatching the bird in mid-air and devouring it in one quick motion.')]"
+      ]
+     },
+     "execution_count": 21,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "%%time\n",
+    "model.batch([prompt, prompt])"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 22,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "CPU times: user 9.72 ms, sys: 7.59 ms, total: 17.3 ms\n",
+      "Wall time: 5.56 s\n"
+     ]
+    }
+   ],
+   "source": [
+    "%%time\n",
+    "for _ in range(2):\n",
+    "    model.invoke(prompt)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 23,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[AIMessage(content=\"\\nSure, here's a three sentence story about a cat:\\n\\nMittens the cat purred contentedly on the windowsill, basking in the warm sunlight that streamed through the glass. Her bright green eyes sparkled as she watched a bird flit and flutter outside, wishing she could join it in its flight. Just then, her owner entered the room with a bowl of creamy milk, causing Mittens to jump down from the windowsill and rub against their legs in excitement.\"),\n",
+       " AIMessage(content='\\nSure! Here is a three sentence story about a cat:\\n\\nMittens the cat purred contentedly on my lap, her soft fur a soothing balm for my frazzled nerves. As I stroked her back, she gazed up at me with big, round eyes, purring even louder. It was hard to resist the charm of this little ball of fluff, and I found myself smiling and scratching behind her ears.')]"
+      ]
+     },
+     "execution_count": 23,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "await model.abatch([prompt, prompt])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Streaming is available by default"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 24,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "|S|ure|,| here| is| a| |3| sentence| story| about| a| cat|:|\n",
+      "|\n",
+      "|M|itt|ens| the| cat| pur|red| content|edly| on| the| windows|ill|,| her| tail| tw|itch|ing| as| she| watched| the| birds| outside|.| Sud|den|ly|,| a| squ|ir|rel| sc|am|per|ed| by| and| Mitt|ens| was| on| high| alert|,| her| ears| per|ked| up| and| ready| to| p|ounce|.| With| light|ning| quick| ref|lex|es|,| Mitt|ens| jump|ed| from| the| windows|ill| and| ch|ased| after| the| squ|ir|rel|,| her| tail| streaming| behind| her|.||"
+     ]
+    }
+   ],
+   "source": [
+    "for chunk in model.stream(prompt):\n",
+    "    print(chunk.content, end=\"|\", flush=True)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 25,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "|The| cat| pur|red| content|edly| on| my| lap|,| its| soft| fur| a| so|othing| bal|m| for| my| fra|zz|led| n|erves|.| As| I| st|rok|ed| its| back|,| it| gaz|ed| up| at| me| with| soul|ful| eyes|,| p|urr|ing| loud|ly| in| appro|val|.| In| that| moment|,| all| was| right| with| the| world|.||"
+     ]
+    }
+   ],
+   "source": [
+    "async for chunk in model.astream(prompt):\n",
+    "    print(chunk.content, end=\"|\", flush=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "And so is the event stream API"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 26,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "{'event': 'on_chat_model_start', 'run_id': '6c2bdfc1-d482-4861-886c-c737a50771c3', 'name': '/ollama', 'tags': [], 'metadata': {}, 'data': {'input': 'Tell me a 3 sentence story about a cat.'}}\n",
+      "{'event': 'on_chat_model_stream', 'run_id': '6c2bdfc1-d482-4861-886c-c737a50771c3', 'tags': [], 'metadata': {}, 'name': '/ollama', 'data': {'chunk': AIMessageChunk(content='\\n')}}\n",
+      "{'event': 'on_chat_model_stream', 'run_id': '6c2bdfc1-d482-4861-886c-c737a50771c3', 'tags': [], 'metadata': {}, 'name': '/ollama', 'data': {'chunk': AIMessageChunk(content='S')}}\n",
+      "{'event': 'on_chat_model_stream', 'run_id': '6c2bdfc1-d482-4861-886c-c737a50771c3', 'tags': [], 'metadata': {}, 'name': '/ollama', 'data': {'chunk': AIMessageChunk(content='ure')}}\n",
+      "{'event': 'on_chat_model_stream', 'run_id': '6c2bdfc1-d482-4861-886c-c737a50771c3', 'tags': [], 'metadata': {}, 'name': '/ollama', 'data': {'chunk': AIMessageChunk(content=',')}}\n",
+      "{'event': 'on_chat_model_stream', 'run_id': '6c2bdfc1-d482-4861-886c-c737a50771c3', 'tags': [], 'metadata': {}, 'name': '/ollama', 'data': {'chunk': AIMessageChunk(content=' here')}}\n",
+      "{'event': 'on_chat_model_stream', 'run_id': '6c2bdfc1-d482-4861-886c-c737a50771c3', 'tags': [], 'metadata': {}, 'name': '/ollama', 'data': {'chunk': AIMessageChunk(content=' is')}}\n",
+      "{'event': 'on_chat_model_stream', 'run_id': '6c2bdfc1-d482-4861-886c-c737a50771c3', 'tags': [], 'metadata': {}, 'name': '/ollama', 'data': {'chunk': AIMessageChunk(content=' a')}}\n",
+      "{'event': 'on_chat_model_stream', 'run_id': '6c2bdfc1-d482-4861-886c-c737a50771c3', 'tags': [], 'metadata': {}, 'name': '/ollama', 'data': {'chunk': AIMessageChunk(content=' ')}}\n",
+      "{'event': 'on_chat_model_stream', 'run_id': '6c2bdfc1-d482-4861-886c-c737a50771c3', 'tags': [], 'metadata': {}, 'name': '/ollama', 'data': {'chunk': AIMessageChunk(content='3')}}\n",
+      "{'event': 'on_chat_model_stream', 'run_id': '6c2bdfc1-d482-4861-886c-c737a50771c3', 'tags': [], 'metadata': {}, 'name': '/ollama', 'data': {'chunk': AIMessageChunk(content=' sentence')}}\n",
+      "{'event': 'on_chat_model_stream', 'run_id': '6c2bdfc1-d482-4861-886c-c737a50771c3', 'tags': [], 'metadata': {}, 'name': '/ollama', 'data': {'chunk': AIMessageChunk(content=' story')}}\n",
+      "...\n"
+     ]
+    }
+   ],
+   "source": [
+    "i = 0\n",
+    "async for event in model.astream_events(prompt, version='v1'):\n",
+    "    print(event)\n",
+    "    if i > 10:\n",
+    "        print('...')\n",
+    "        break\n",
+    "    i += 1"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.7"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

examples/local_llm/server.py

@@ -0,0 +1,38 @@
+#!/usr/bin/env python
+"""Example LangChain Server that runs a local llm.
+
+**Attention** This is OK for prototyping / dev usage, but should not be used
+for production cases when there might be concurrent requests from different
+users. As of the time of writing, Ollama is designed for single user and cannot
+handle concurrent requests see this issue: 
+https://github.com/ollama/ollama/issues/358
+
+When deploying local models, make sure you understand whether the model is able
+to handle concurrent requests or not. If concurrent requests are not handled
+properly, the server will either crash or will just not be able to handle more
+than one user at a time.
+"""
+from fastapi import FastAPI
+from langchain_community.chat_models import ChatOllama
+
+from langserve import add_routes
+
+app = FastAPI(
+    title="LangChain Server",
+    version="1.0",
+    description="Spin up a simple api server using Langchain's Runnable interfaces",
+)
+
+
+llm = ChatOllama(model="llama2")
+
+add_routes(
+    app,
+    llm,
+    path="/ollama",
+)
+
+if __name__ == "__main__":
+    import uvicorn
+
+    uvicorn.run(app, host="localhost", port=8000)

examples/multiple_servers/client_server.py

@@ -0,0 +1,35 @@
+"""Client server that interacts with the main server via a remote runnable.
+
+This server sets up a simple proxy to the main server. It uses the RemoteRunnable
+to interact with the main server. The main server is expected to be running at
+http://localhost:8123.
+
+A client server will likely end up doing something more clever rather than
+just being a proxy.
+"""
+from fastapi import FastAPI
+
+from langserve import RemoteRunnable, add_routes
+
+app = FastAPI()
+
+MAIN_SERVER_URL = (
+    "http://localhost:8123/chat_model/"  # <-- URL of the RUNNABLE on the main server
+)
+# Type inference is not automatic for remote runnables at the moment,
+# so you must specify which types are used for the playground to work.
+remote_runnable = RemoteRunnable(MAIN_SERVER_URL).with_types(input_type=str)
+
+
+# Let's add an example chain
+add_routes(
+    app,
+    remote_runnable,
+    path="/proxied",
+)
+
+
+if __name__ == "__main__":
+    import uvicorn
+
+    uvicorn.run(app)

examples/multiple_servers/main_server.py

@@ -0,0 +1,20 @@
+"""Main server that exposes one or more chains as HTTP endpoints."""
+from fastapi import FastAPI
+from langchain_openai import ChatOpenAI
+
+from langserve import add_routes
+
+app = FastAPI()
+
+# Let's add an example chain
+add_routes(
+    app,
+    ChatOpenAI(),
+    path="/chat_model",
+)
+
+if __name__ == "__main__":
+    import uvicorn
+
+    # Running on port 8123
+    uvicorn.run(app, port=8123)

examples/passthrough_dict/client.ipynb

@@ -0,0 +1,148 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Passthrough information\n",
+    "\n",
+    "An example that shows how to pass through additional info with the request, and get it back with the response."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "from langchain.prompts.chat import ChatPromptTemplate"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "from langserve import RemoteRunnable\n",
+    "\n",
+    "chain = RemoteRunnable(\"http://localhost:8000/v1/\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Let's create a prompt composed of a system message and a human message."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'output': AIMessage(content='`apple` translates to `mela` in Italian.'),\n",
+       " 'info': {'info': {'user_id': 42, 'user_info': {'address': 42}}}}"
+      ]
+     },
+     "execution_count": 10,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "chain.invoke({'thing': 'apple', 'language': 'italian', 'info': {\"user_id\": 42, \"user_info\": {\"address\": 42}}})"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "{'info': {'info': {'user_id': 42, 'user_info': {'address': 42}}}}\n",
+      "{'output': AIMessageChunk(content='')}\n",
+      "{'output': AIMessageChunk(content='m')}\n",
+      "{'output': AIMessageChunk(content='ela')}\n",
+      "{'output': AIMessageChunk(content='')}\n"
+     ]
+    }
+   ],
+   "source": [
+    "for chunk in chain.stream({'thing': 'apple', 'language': 'italian', 'info': {\"user_id\": 42, \"user_info\": {\"address\": 42}}}):\n",
+    "    print(chunk)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "from langserve import RemoteRunnable\n",
+    "\n",
+    "chain = RemoteRunnable(\"http://localhost:8000/v2/\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'output': AIMessage(content='`apple` translates to `mela` in Italian.'),\n",
+       " 'info': {'user_id': 42, 'user_info': {'address': 42}}}"
+      ]
+     },
+     "execution_count": 13,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "chain.invoke({'thing': 'apple', 'language': 'italian', 'info': {\"user_id\": 42, \"user_info\": {\"address\": 42}}})"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.6"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

examples/passthrough_dict/server.py

@@ -0,0 +1,86 @@
+#!/usr/bin/env python
+"""Example LangChain server passes through some of the inputs in the response."""
+
+from typing import Any, Callable, Dict, List, Optional, TypedDict
+
+from fastapi import FastAPI
+from langchain.chat_models import ChatOpenAI
+from langchain.prompts import ChatPromptTemplate
+from langchain.schema.runnable import RunnableMap, RunnablePassthrough
+
+from langserve import add_routes
+
+app = FastAPI(
+    title="LangChain Server",
+    version="1.0",
+    description="Spin up a simple api server using Langchain's Runnable interfaces",
+)
+
+
+def _create_projection(
+    *, include_keys: Optional[List] = None, exclude_keys: Optional[List[str]] = None
+) -> Callable[[dict], dict]:
+    """Create a projection function."""
+
+    def _project_dict(
+        d: dict,
+    ) -> dict:
+        """Project dictionary."""
+        keys = d.keys()
+        if include_keys is not None:
+            keys = set(keys) & set(include_keys)
+        if exclude_keys is not None:
+            keys = set(keys) - set(exclude_keys)
+        return {k: d[k] for k in keys}
+
+    return _project_dict
+
+
+prompt = ChatPromptTemplate.from_messages(
+    [("human", "translate `{thing}` to {language}")]
+)
+model = ChatOpenAI()
+
+underlying_chain = prompt | model
+
+wrapped_chain = RunnableMap(
+    {
+        "output": _create_projection(exclude_keys=["info"]) | underlying_chain,
+        "info": _create_projection(include_keys=["info"]),
+    }
+)
+
+
+class Input(TypedDict):
+    thing: str
+    language: str
+    info: Dict[str, Any]
+
+
+class Output(TypedDict):
+    output: underlying_chain.output_schema
+    info: Dict[str, Any]
+
+
+add_routes(
+    app, wrapped_chain.with_types(input_type=Input, output_type=Output), path="/v1"
+)
+
+
+# Version 2
+# Uses RunnablePassthrough.assign
+wrapped_chain_2 = RunnablePassthrough.assign(output=underlying_chain) | {
+    "output": lambda x: x["output"],
+    "info": lambda x: x["info"],
+}
+
+add_routes(
+    app,
+    wrapped_chain_2.with_types(input_type=Input, output_type=Output),
+    path="/v2",
+)
+
+if __name__ == "__main__":
+    import uvicorn
+
+    uvicorn.run(app, host="localhost", port=8000)

examples/router/server.py

@@ -0,0 +1,38 @@
+#!/usr/bin/env python
+"""Example LangChain Server that uses a Fast API Router.
+
+When applications grow, it becomes useful to use FastAPI's Router to organize
+the routes.
+
+See more documentation at:
+
+https://fastapi.tiangolo.com/tutorial/bigger-applications/
+"""
+from fastapi import APIRouter, FastAPI
+from langchain.chat_models import ChatAnthropic, ChatOpenAI
+
+from langserve import add_routes
+
+app = FastAPI()
+
+router = APIRouter(prefix="/models")
+
+# Invocations to this router will appear in trace logs as /models/openai
+add_routes(
+    router,
+    ChatOpenAI(),
+    path="/openai",
+)
+# Invocations to this router will appear in trace logs as /models/anthropic
+add_routes(
+    router,
+    ChatAnthropic(),
+    path="/anthropic",
+)
+
+app.include_router(router)
+
+if __name__ == "__main__":
+    import uvicorn
+
+    uvicorn.run(app, host="localhost", port=8000)

examples/widgets/server.py

@@ -6,6 +6,7 @@ from typing import Any, Dict, List, Tuple
 
 from fastapi import FastAPI
 from fastapi.middleware.cors import CORSMiddleware
+from langchain.chat_models.openai import ChatOpenAI
 from langchain.document_loaders.blob_loaders import Blob
 from langchain.document_loaders.parsers.pdf import PDFMinerParser
 from langchain.pydantic_v1 import BaseModel, Field
@@ -13,9 +14,12 @@ from langchain.schema.messages import (
     AIMessage,
     BaseMessage,
     FunctionMessage,
+    HumanMessage,
 )
 from langchain.schema.runnable import RunnableLambda
+from langchain_core.runnables import RunnableParallel
 
+from langserve import CustomUserType
 from langserve.server import add_routes
 
 app = FastAPI(
@@ -35,8 +39,11 @@ app.add_middleware(
     expose_headers=["*"],
 )
 
+# Example 1: Chat Widget
+# This shows how to create a chat widget.
 
-class ChatHistory(BaseModel):
+
+class ChatHistory(CustomUserType):
     chat_history: List[Tuple[str, str]] = Field(
         ...,
         examples=[[("a", "aa")]],
@@ -45,27 +52,43 @@ class ChatHistory(BaseModel):
     question: str
 
 
+def _format_to_messages(input: ChatHistory) -> List[BaseMessage]:
+    """Format the input to a list of messages."""
+    history = input.chat_history
+    user_input = input.question
+
+    messages = []
+
+    for human, ai in history:
+        messages.append(HumanMessage(content=human))
+        messages.append(AIMessage(content=ai))
+    messages.append(HumanMessage(content=user_input))
+    return messages
+
+
+model = ChatOpenAI()
+chat_model = RunnableParallel({"answer": (RunnableLambda(_format_to_messages) | model)})
+add_routes(
+    app,
+    chat_model.with_types(input_type=ChatHistory),
+    config_keys=["configurable"],
+    path="/chat",
+)
+
+
+# Example 2: Chat Widget with History
+# This one isn't hooked up toa model. It just shows that FunctionMessages can be used
+# surfaced as well in the playground.
+
+
 class ChatHistoryMessage(BaseModel):
     chat_history: List[BaseMessage] = Field(
         ...,
-        extra={"widget": {"type": "chat", "input": "location", "output": "output"}},
+        extra={"widget": {"type": "chat", "input": "location"}},
     )
     location: str
 
 
-class FileProcessingRequest(BaseModel):
-    file: bytes = Field(..., extra={"widget": {"type": "base64file"}})
-    num_chars: int = 100
-
-
-def chat_with_bot(input: Dict[str, Any]) -> Dict[str, Any]:
-    """Bot that repeats the question twice."""
-    return {
-        "answer": input["question"] * 2,
-        "woof": "its so bad to woof, meow is better",
-    }
-
-
 def chat_message_bot(input: Dict[str, Any]) -> List[BaseMessage]:
     """Bot that repeats the question twice."""
     return [
@@ -83,6 +106,21 @@ def chat_message_bot(input: Dict[str, Any]) -> List[BaseMessage]:
     ]
 
 
+add_routes(
+    app,
+    RunnableLambda(chat_message_bot).with_types(input_type=ChatHistoryMessage),
+    config_keys=["configurable"],
+    path="/chat_message",
+)
+
+# Example 3: File Processing Widget
+
+
+class FileProcessingRequest(BaseModel):
+    file: bytes = Field(..., extra={"widget": {"type": "base64file"}})
+    num_chars: int = 100
+
+
 def process_file(input: Dict[str, Any]) -> str:
     """Extract the text from the first page of the PDF."""
     content = base64.decodebytes(input["file"])
@@ -92,13 +130,6 @@ def process_file(input: Dict[str, Any]) -> str:
     return content[: input["num_chars"]]
 
 
-add_routes(
-    app,
-    RunnableLambda(chat_with_bot).with_types(input_type=ChatHistory),
-    config_keys=["configurable"],
-    path="/chat",
-)
-
 add_routes(
     app,
     RunnableLambda(process_file).with_types(input_type=FileProcessingRequest),
@@ -106,14 +137,6 @@ add_routes(
     path="/pdf",
 )
 
-add_routes(
-    app,
-    RunnableLambda(chat_message_bot).with_types(input_type=ChatHistoryMessage),
-    config_keys=["configurable"],
-    path="/chat_message",
-)
-
-
 if __name__ == "__main__":
     import uvicorn
 

langserve/__init__.py

@@ -4,9 +4,16 @@ This is the ONLY public interface into the package. All other modules are
 to be considered private and subject to change without notice.
 """
 
+from langserve.api_handler import APIHandler
 from langserve.client import RemoteRunnable
 from langserve.schema import CustomUserType
 from langserve.server import add_routes
 from langserve.version import __version__
 
-__all__ = ["RemoteRunnable", "add_routes", "__version__", "CustomUserType"]
+__all__ = [
+    "RemoteRunnable",
+    "APIHandler",
+    "add_routes",
+    "__version__",
+    "CustomUserType",
+]

langserve/client.py

@@ -36,6 +36,8 @@ from langchain.schema.runnable.config import (
     get_callback_manager_for_config,
 )
 from langchain.schema.runnable.utils import AddableDict, Input, Output
+from langchain_core.runnables.schema import StreamEvent
+from typing_extensions import Literal
 
 from langserve.callbacks import CallbackEventDict, ahandle_callbacks, handle_callbacks
 from langserve.serialization import (
@@ -47,10 +49,68 @@ from langserve.serialization import (
 logger = logging.getLogger(__name__)
 
 
-def _without_callbacks(config: Optional[RunnableConfig]) -> RunnableConfig:
-    """Evict callbacks from the config since those are definitely not supported."""
+def _is_json_serializable(obj: Any) -> bool:
+    """Return True if the object is json serializable."""
+    if isinstance(obj, (tuple, list, dict, str, int, float, bool, type(None))):
+        return True
+    else:
+        return False
+
+
+def _keep_json_serializable(obj: Any) -> Any:
+    """Traverse the object recursively and removes non-json serializable objects."""
+    if isinstance(obj, dict):
+        return {
+            k: _keep_json_serializable(v)
+            for k, v in obj.items()
+            if isinstance(k, str) and _is_json_serializable(v)
+        }
+    elif isinstance(obj, (list, tuple)):
+        return [_keep_json_serializable(v) for v in obj if _is_json_serializable(v)]
+    elif _is_json_serializable(obj):
+        return obj
+    else:
+        raise AssertionError("This code should not be reachable. If it's reached")
+
+
+def _prepare_config_for_server(
+    config: Optional[RunnableConfig], *, ignore_unserializable: bool = True
+) -> RunnableConfig:
+    """Evict information from the config that should not be sent to the server.
+
+    This includes:
+    - callbacks: Callbacks are handled separately
+    - non-json serializable objects: We cannot serialize then the correct behavior
+        these appear frequently in the config of the runnable but are only needed
+        in the local scope of the config (they do not need to be sent to the server).
+        An example are the write / read channel objects populated by langgraph,
+        or the 'messages' field in configurable populated by RunnableWithMessageHistory.
+
+    Args:
+        config: The config to clean up
+        ignore_unserializable: If True, will ignore non-json serializable objects
+            found in the 'configurable' field of the config.
+            This is expected to be the safe default to use since the server
+            should not be specifying configurable objects that are not json
+            serializable. This logic is expected mostly to with non serializable
+            content that was created for local use by the runnable, and
+            is not needed by the server.
+            If False, will raise an error if a non-json serializable object is found.
+
+    Returns:
+        A cleaned up version of the config that can be sent to the server.
+    """
     _config = config or {}
-    return {k: v for k, v in _config.items() if k != "callbacks"}
+    without_callbacks = {k: v for k, v in _config.items() if k != "callbacks"}
+    if "configurable" in without_callbacks:
+        # Get a version of
+
+        if ignore_unserializable:
+            without_callbacks["configurable"] = _keep_json_serializable(
+                without_callbacks["configurable"]
+            )
+
+    return without_callbacks
 
 
 @lru_cache(maxsize=1_000)  # Will accommodate up to 1_000 different error messages
@@ -235,7 +295,8 @@ class RemoteRunnable(Runnable[Input, Output]):
                 callback events returned by the server.
         """
         _client_kwargs = client_kwargs or {}
-        self.url = url
+        # Enforce trailing slash
+        self.url = url if url.endswith("/") else url + "/"
         self.sync_client = httpx.Client(
             base_url=url,
             timeout=timeout,
@@ -274,7 +335,7 @@ class RemoteRunnable(Runnable[Input, Output]):
             "/invoke",
             json={
                 "input": self._lc_serializer.dumpd(input),
-                "config": _without_callbacks(config),
+                "config": _prepare_config_for_server(config),
                 "kwargs": kwargs,
             },
         )
@@ -305,7 +366,7 @@ class RemoteRunnable(Runnable[Input, Output]):
             "/invoke",
             json={
                 "input": self._lc_serializer.dumpd(input),
-                "config": _without_callbacks(config),
+                "config": _prepare_config_for_server(config),
                 "kwargs": kwargs,
             },
         )
@@ -340,9 +401,9 @@ class RemoteRunnable(Runnable[Input, Output]):
             )
 
         if isinstance(config, list):
-            _config = [_without_callbacks(c) for c in config]
+            _config = [_prepare_config_for_server(c) for c in config]
         else:
-            _config = _without_callbacks(config)
+            _config = _prepare_config_for_server(config)
 
         response = self.sync_client.post(
             "/batch",
@@ -365,11 +426,6 @@ class RemoteRunnable(Runnable[Input, Output]):
 
         return outputs
 
-    def _enforce_trailing_slash(self, url: str) -> str:
-        if url.endswith("/"):
-            return url
-        return url + "/"
-
     def batch(
         self,
         inputs: List[Input],
@@ -398,9 +454,9 @@ class RemoteRunnable(Runnable[Input, Output]):
             )
 
         if isinstance(config, list):
-            _config = [_without_callbacks(c) for c in config]
+            _config = [_prepare_config_for_server(c) for c in config]
         else:
-            _config = _without_callbacks(config)
+            _config = _prepare_config_for_server(config)
 
         response = await self.async_client.post(
             "/batch",
@@ -462,10 +518,10 @@ class RemoteRunnable(Runnable[Input, Output]):
         )
         data = {
             "input": self._lc_serializer.dumpd(input),
-            "config": _without_callbacks(config),
+            "config": _prepare_config_for_server(config),
             "kwargs": kwargs,
         }
-        endpoint = urljoin(self._enforce_trailing_slash(self.url), "stream")
+        endpoint = urljoin(self.url, "stream")
 
         try:
             from httpx_sse import connect_sse
@@ -512,10 +568,13 @@ class RemoteRunnable(Runnable[Input, Output]):
                         _raise_exception_from_data(
                             sse.data, httpx.Request(method="POST", url=endpoint)
                         )
+                    elif sse.event == "metadata":
+                        # Nothing to do for metadata for the regular remote client.
+                        continue
                     elif sse.event == "end":
                         break
                     else:
-                        logger.error(
+                        _log_error_message_once(
                             f"Encountered an unsupported event type: `{sse.event}`. "
                             f"Try upgrading the remote client to the latest version."
                             f"Ignoring events of type `{sse.event}`."
@@ -545,10 +604,10 @@ class RemoteRunnable(Runnable[Input, Output]):
         )
         data = {
             "input": self._lc_serializer.dumpd(input),
-            "config": _without_callbacks(config),
+            "config": _prepare_config_for_server(config),
             "kwargs": kwargs,
         }
-        endpoint = urljoin(self._enforce_trailing_slash(self.url), "stream")
+        endpoint = urljoin(self.url, "stream")
 
         try:
             from httpx_sse import aconnect_sse
@@ -593,10 +652,13 @@ class RemoteRunnable(Runnable[Input, Output]):
                         _raise_exception_from_data(
                             sse.data, httpx.Request(method="POST", url=endpoint)
                         )
+                    elif sse.event == "metadata":
+                        # Nothing to do for metadata for the regular remote client.
+                        continue
                     elif sse.event == "end":
                         break
                     else:
-                        logger.error(
+                        _log_error_message_once(
                             f"Encountered an unsupported event type: `{sse.event}`. "
                             f"Try upgrading the remote client to the latest version."
                             f"Ignoring events of type `{sse.event}`."
@@ -644,7 +706,7 @@ class RemoteRunnable(Runnable[Input, Output]):
         )
         data = {
             "input": self._lc_serializer.dumpd(input),
-            "config": _without_callbacks(config),
+            "config": _prepare_config_for_server(config),
             "kwargs": kwargs,
             "diff": True,
             "include_names": include_names,
@@ -668,9 +730,13 @@ class RemoteRunnable(Runnable[Input, Output]):
                 async for sse in event_source.aiter_sse():
                     if sse.event == "data":
                         data = self._lc_serializer.loads(sse.data)
+                        # Create a copy of the data to yield since underlying
+                        # code is using jsonpatch which does some stuff in-place
+                        # that can cause unexpected consequences.
+                        chunk_to_yield = RunLogPatch(*copy.deepcopy(data["ops"]))
                         chunk = RunLogPatch(*data["ops"])
 
-                        yield chunk
+                        yield chunk_to_yield
                         if final_output:
                             final_output += chunk
                         else:
@@ -693,3 +759,102 @@ class RemoteRunnable(Runnable[Input, Output]):
             raise
         else:
             await run_manager.on_chain_end(final_output)
+
+    async def astream_events(
+        self,
+        input: Any,
+        config: Optional[RunnableConfig] = None,
+        *,
+        version: Literal["v1"],
+        include_names: Optional[Sequence[str]] = None,
+        include_types: Optional[Sequence[str]] = None,
+        include_tags: Optional[Sequence[str]] = None,
+        exclude_names: Optional[Sequence[str]] = None,
+        exclude_types: Optional[Sequence[str]] = None,
+        exclude_tags: Optional[Sequence[str]] = None,
+        **kwargs: Any,
+    ) -> AsyncIterator[StreamEvent]:
+        """Stream events from the server runnable.
+
+        **Attention**: This method is using a beta API and may change slightly.
+
+        This method can stream events from any step used in the runnable exposed
+        on the server. This includes all inner runs of LLMs, Retrievers, Tools, etc.
+
+        **Recommended**: Only ask for the data you need. This can significantly
+        reduce the amount of data sent over the wire.
+
+        Args:
+            input: The input to the runnable
+            config: The config to use for the runnable
+            version: The version of the astream_events to use.
+                     Currently only "v1" is supported.
+            include_names: The names of the events to include
+            include_types: The types of the events to include
+            include_tags: The tags of the events to include
+            exclude_names: The names of the events to exclude
+            exclude_types: The types of the events to exclude
+            exclude_tags: The tags of the events to exclude
+        """
+        if version != "v1":
+            raise ValueError(f"Unsupported version: {version}. Use 'v1'")
+
+        # Create a stream handler that will emit Log objects
+        config = ensure_config(config)
+        callback_manager = get_async_callback_manager_for_config(config)
+
+        events = []
+
+        run_manager = await callback_manager.on_chain_start(
+            dumpd(self),
+            self._lc_serializer.dumpd(input),
+            name=config.get("run_name"),
+        )
+        data = {
+            "input": self._lc_serializer.dumpd(input),
+            "config": _prepare_config_for_server(config),
+            "kwargs": kwargs,
+            "include_names": include_names,
+            "include_types": include_types,
+            "include_tags": include_tags,
+            "exclude_names": exclude_names,
+            "exclude_types": exclude_types,
+            "exclude_tags": exclude_tags,
+        }
+        endpoint = urljoin(self.url, "stream_events")
+
+        try:
+            from httpx_sse import aconnect_sse
+        except ImportError:
+            raise ImportError("You must install `httpx_sse` to use the stream method.")
+
+        try:
+            async with aconnect_sse(
+                self.async_client, "POST", endpoint, json=data
+            ) as event_source:
+                async for sse in event_source.aiter_sse():
+                    if sse.event == "data":
+                        event = self._lc_serializer.loads(sse.data)
+                        # Create a copy of the data to yield since underlying
+                        # code is using jsonpatch which does some stuff in-place
+                        # that can cause unexpected consequences.
+                        yield event
+                        events.append(event)
+                    elif sse.event == "error":
+                        # This can only be a server side error
+                        _raise_exception_from_data(
+                            sse.data, httpx.Request(method="POST", url=endpoint)
+                        )
+                    elif sse.event == "end":
+                        break
+                    else:
+                        _log_error_message_once(
+                            f"Encountered an unsupported event type: `{sse.event}`. "
+                            f"Try upgrading the remote client to the latest version."
+                            f"Ignoring events of type `{sse.event}`."
+                        )
+        except BaseException as e:
+            await run_manager.on_chain_error(e)
+            raise
+        else:
+            await run_manager.on_chain_end(events)

langserve/playground.py

@@ -7,22 +7,53 @@ from typing import Sequence, Type
 from fastapi.responses import Response
 from langchain.schema.runnable import Runnable
 
-try:
-    from pydantic.v1 import BaseModel
-except ImportError:
-    from pydantic import BaseModel
+from langserve.pydantic_v1 import BaseModel
 
 
 class PlaygroundTemplate(Template):
     delimiter = "____"
 
 
+def _get_mimetype(path: str) -> str:
+    """Get mimetype for file.
+
+    Custom implementation of mimetypes.guess_type that
+    uses the file extension to determine the mimetype for some files.
+
+    This is necessary due to: https://bugs.python.org/issue43975
+    Resolves issue: https://github.com/langchain-ai/langserve/issues/245
+
+    Args:
+        path (str): Path to file
+
+    Returns:
+        str: Mimetype of file
+    """
+    try:
+        file_extension = path.lower().split(".")[-1]
+    except IndexError:
+        return mimetypes.guess_type(path)[0]
+
+    if file_extension == "js":
+        return "application/javascript"
+    elif file_extension == "css":
+        return "text/css"
+    elif file_extension in ["htm", "html"]:
+        return "text/html"
+
+    # If the file extension is not one of the specified ones,
+    # use the default guess method
+    mime_type = mimetypes.guess_type(path)[0]
+    return mime_type
+
+
 async def serve_playground(
     runnable: Runnable,
     input_schema: Type[BaseModel],
     config_keys: Sequence[str],
     base_url: str,
     file_path: str,
+    feedback_enabled: bool,
 ) -> Response:
     """Serve the playground."""
     local_file_path = os.path.abspath(
@@ -42,7 +73,7 @@ async def serve_playground(
 
     try:
         with open(local_file_path, encoding="utf-8") as f:
-            mime_type = mimetypes.guess_type(local_file_path)[0]
+            mime_type = _get_mimetype(local_file_path)
             if mime_type in ("text/html", "text/css", "application/javascript"):
                 response = PlaygroundTemplate(f.read()).substitute(
                     LANGSERVE_BASE_URL=base_url[1:]
@@ -52,6 +83,9 @@ async def serve_playground(
                         runnable.config_schema(include=config_keys).schema()
                     ),
                     LANGSERVE_INPUT_SCHEMA=json.dumps(input_schema.schema()),
+                    LANGSERVE_FEEDBACK_ENABLED=json.dumps(
+                        "true" if feedback_enabled else "false"
+                    ),
                 )
             else:
                 response = f.buffer.read()

langserve/playground/dist/index.html

@@ -5,8 +5,8 @@
     <link rel="icon" href="/____LANGSERVE_BASE_URL/favicon.ico" />
     <meta name="viewport" content="width=device-width, initial-scale=1.0" />
     <title>Playground</title>
-    <script type="module" crossorigin src="/____LANGSERVE_BASE_URL/assets/index-7c3e1e1d.js"></script>
-    <link rel="stylesheet" href="/____LANGSERVE_BASE_URL/assets/index-c6cde0dd.css">
+    <script type="module" crossorigin src="/____LANGSERVE_BASE_URL/assets/index-fbe3df33.js"></script>
+    <link rel="stylesheet" href="/____LANGSERVE_BASE_URL/assets/index-52e8ab2f.css">
   </head>
   <body>
     <div id="root"></div>
@@ -14,6 +14,7 @@
       try {
         window.CONFIG_SCHEMA = ____LANGSERVE_CONFIG_SCHEMA;
         window.INPUT_SCHEMA = ____LANGSERVE_INPUT_SCHEMA;
+        window.FEEDBACK_ENABLED = ____LANGSERVE_FEEDBACK_ENABLED;
       } catch (error) {
         // pass
       }

langserve/playground/index.html

@@ -12,6 +12,7 @@
       try {
         window.CONFIG_SCHEMA = ____LANGSERVE_CONFIG_SCHEMA;
         window.INPUT_SCHEMA = ____LANGSERVE_INPUT_SCHEMA;
+        window.FEEDBACK_ENABLED = ____LANGSERVE_FEEDBACK_ENABLED;
       } catch (error) {
         // pass
       }

langserve/playground/package.json

@@ -28,6 +28,7 @@
     "lz-string": "^1.5.0",
     "react": "^18.2.0",
     "react-dom": "^18.2.0",
+    "swr": "^2.2.4",
     "tailwind-merge": "^1.14.0",
     "use-debounce": "^9.0.4",
     "vaul": "^0.7.3"

langserve/playground/src/App.tsx

@@ -1,258 +1,55 @@
 import "./App.css";
 
-import { useEffect, useMemo, useRef, useState } from "react";
-import defaults from "./utils/defaults";
-import { JsonForms } from "@jsonforms/react";
-import {
-  materialAllOfControlTester,
-  MaterialAllOfRenderer,
-  MaterialObjectRenderer,
-  materialOneOfControlTester,
-  MaterialOneOfRenderer,
-} from "@jsonforms/material-renderers";
-import dayjs from "dayjs";
-import utc from "dayjs/plugin/utc";
-import relativeDate from "dayjs/plugin/relativeTime";
-import SendIcon from "./assets/SendIcon.svg?react";
-import ShareIcon from "./assets/ShareIcon.svg?react";
-import { compressToEncodedURIComponent } from "lz-string";
+import { useEffect, useRef, useState } from "react";
 
-import {
-  BooleanCell,
-  DateCell,
-  DateTimeCell,
-  EnumCell,
-  IntegerCell,
-  NumberCell,
-  SliderCell,
-  TimeCell,
-  booleanCellTester,
-  dateCellTester,
-  dateTimeCellTester,
-  enumCellTester,
-  integerCellTester,
-  numberCellTester,
-  sliderCellTester,
-  textAreaCellTester,
-  textCellTester,
-  timeCellTester,
-  vanillaRenderers,
-  InputControl,
-} from "@jsonforms/vanilla-renderers";
-import { useSchemas } from "./useSchemas";
+import ShareIcon from "./assets/ShareIcon.svg?react";
+
+import { useConfigSchema, useFeedback, useInputSchema } from "./useSchemas";
 import { useStreamLog } from "./useStreamLog";
-import { StreamCallback } from "./types";
-import { AppCallbackContext } from "./useStreamCallback";
-import {
-  JsonFormsCore,
-  RankedTester,
-  rankWith,
-  and,
-  uiTypeIs,
-  schemaMatches,
-  schemaTypeIs,
-} from "@jsonforms/core";
-import CustomArrayControlRenderer, {
-  materialArrayControlTester,
-} from "./components/CustomArrayControlRenderer";
-import CustomTextAreaCell from "./components/CustomTextAreaCell";
-import JsonTextAreaCell from "./components/JsonTextAreaCell";
-import { getStateFromUrl, ShareDialog } from "./components/ShareDialog";
-import {
-  chatMessagesTester,
-  ChatMessagesControlRenderer,
-} from "./components/ChatMessagesControlRenderer";
-import {
-  ChatMessageTuplesControlRenderer,
-  chatMessagesTupleTester,
-} from "./components/ChatMessageTuplesControlRenderer";
-import {
-  fileBase64Tester,
-  FileBase64ControlRenderer,
-} from "./components/FileBase64Tester";
+import { AppCallbackContext, useAppStreamCallbacks } from "./useStreamCallback";
+import { JsonSchema } from "@jsonforms/core";
+import { ShareDialog } from "./components/ShareDialog";
 import { IntermediateSteps } from "./components/IntermediateSteps";
 import { StreamOutput } from "./components/StreamOutput";
-import {
-  customAnyOfTester,
-  CustomAnyOfRenderer,
-} from "./components/CustomAnyOfRenderer";
+import { ConfigValue, SectionConfigure } from "./sections/SectionConfigure";
+import { InputValue, SectionInputs } from "./sections/SectionInputs";
+import { SubmitButton } from "./components/SubmitButton";
+import { useDebounce } from "use-debounce";
 import { cn } from "./utils/cn";
+import { CorrectnessFeedback } from "./components/feedback/CorrectnessFeedback";
+import { getStateFromUrl } from "./utils/url";
 
-dayjs.extend(relativeDate);
-dayjs.extend(utc);
+function InputPlayground(props: {
+  configSchema: { schema: JsonSchema; defaults: unknown };
+  inputSchema: { schema: JsonSchema; defaults: unknown };
 
-const isObjectWithPropertiesControl = rankWith(
-  2,
-  and(
-    uiTypeIs("Control"),
-    schemaTypeIs("object"),
-    schemaMatches((schema) =>
-      Object.prototype.hasOwnProperty.call(schema, "properties")
-    )
-  )
-);
+  configData: ConfigValue;
 
-const isObject = rankWith(1, and(uiTypeIs("Control"), schemaTypeIs("object")));
-const isElse = rankWith(1, and(uiTypeIs("Control")));
+  startStream: (input: unknown, config: unknown) => void;
+  stopStream: (() => void) | undefined;
 
-export const renderers = [
-  ...vanillaRenderers,
-
-  // use material renderers to handle objects and json schema references
-  // they should yield the rendering to simpler cells
-  { tester: isObjectWithPropertiesControl, renderer: MaterialObjectRenderer },
-  { tester: materialAllOfControlTester, renderer: MaterialAllOfRenderer },
-  { tester: materialOneOfControlTester, renderer: MaterialOneOfRenderer },
-
-  { tester: customAnyOfTester, renderer: CustomAnyOfRenderer },
-
-  // custom renderers
-  { tester: materialArrayControlTester, renderer: CustomArrayControlRenderer },
-  { tester: isObject, renderer: InputControl },
-  { tester: chatMessagesTester, renderer: ChatMessagesControlRenderer },
-  {
-    tester: chatMessagesTupleTester,
-    renderer: ChatMessageTuplesControlRenderer,
-  },
-  { tester: fileBase64Tester, renderer: FileBase64ControlRenderer },
-];
-
-const nestedArrayControlTester: RankedTester = rankWith(1, (_, jsonSchema) => {
-  return jsonSchema.type === "array";
-});
-
-export const cells = [
-  { tester: booleanCellTester, cell: BooleanCell },
-  { tester: dateCellTester, cell: DateCell },
-  { tester: dateTimeCellTester, cell: DateTimeCell },
-  { tester: enumCellTester, cell: EnumCell },
-  { tester: integerCellTester, cell: IntegerCell },
-  { tester: numberCellTester, cell: NumberCell },
-  { tester: sliderCellTester, cell: SliderCell },
-  { tester: textAreaCellTester, cell: CustomTextAreaCell },
-  { tester: textCellTester, cell: CustomTextAreaCell },
-  { tester: timeCellTester, cell: TimeCell },
-  { tester: nestedArrayControlTester, cell: CustomArrayControlRenderer },
-  { tester: isElse, cell: JsonTextAreaCell },
-];
-
-function App() {
-  const [isIframe] = useState(() => window.self !== window.top);
-
-  // it is possible that defaults are being applied _after_
-  // the initial update message has been sent from the parent window
-  // so we store the initial config data in a ref
-  const initConfigData = useRef<JsonFormsCore["data"]>(null);
-
-  // store form state
-  const [configData, setConfigData] = useState<
-    Pick<JsonFormsCore, "data" | "errors"> & { defaults: boolean }
-  >({ data: {}, errors: [], defaults: true });
-
-  const [inputData, setInputData] = useState<
-    Pick<JsonFormsCore, "data" | "errors">
-  >({ data: null, errors: [] });
-  // fetch input and config schemas from the server
-  const schemas = useSchemas(configData);
-  // apply defaults defined in each schema
-  useEffect(() => {
-    if (schemas.config) {
-      const state = getStateFromUrl(window.location.href);
-      setConfigData({
-        data:
-          state.configFromUrl ??
-          initConfigData.current ??
-          defaults(schemas.config),
-        errors: [],
-        defaults: true,
-      });
-
-      setInputData({ data: defaults(schemas.input), errors: [] });
-    }
-    // eslint-disable-next-line react-hooks/exhaustive-deps
-  }, [schemas.config]);
-
-  // callbacks handling
-  const callbacks = useRef<{
-    onStart: Exclude<StreamCallback["onStart"], undefined>[];
-    onSuccess: Exclude<StreamCallback["onSuccess"], undefined>[];
-    onError: Exclude<StreamCallback["onError"], undefined>[];
-  }>({ onStart: [], onSuccess: [], onError: [] });
-
-  // the runner
-  const { startStream, stopStream, latest } = useStreamLog({
-    onStart(...args) {
-      for (const callback of callbacks.current.onStart) {
-        callback(...args);
-      }
-    },
-    onSuccess(...args) {
-      for (const callback of callbacks.current.onSuccess) {
-        callback(...args);
-      }
-    },
-    onError(...args) {
-      for (const callback of callbacks.current.onError) {
-        callback(...args);
-      }
-    },
+  children?: React.ReactNode;
+}) {
+  const [inputData, setInputData] = useState<InputValue>({
+    data: props.inputSchema.defaults,
+    errors: [],
   });
 
-  useEffect(() => {
-    window.parent?.postMessage({ type: "init" }, "*");
-  }, []);
-
-  useEffect(() => {
-    function listener(event: MessageEvent) {
-      if (event.source === window.parent) {
-        const message = event.data;
-        if (typeof message === "object" && message != null) {
-          switch (message.type) {
-            case "update": {
-              const value: { config: JsonFormsCore["data"] } = message.value;
-              if (Object.keys(value.config).length > 0) {
-                initConfigData.current = value.config;
-                setConfigData({
-                  data: value.config,
-                  errors: [],
-                  defaults: false,
-                });
-                break;
-              }
-            }
-          }
-        }
-      }
-    }
-
-    window.addEventListener("message", listener);
-    return () => window.removeEventListener("message", listener);
-  }, []);
-
-  const isInputResetable = useMemo(() => {
-    if (!schemas.input) return false;
-    return (
-      JSON.stringify(defaults(schemas.input)) !== JSON.stringify(inputData.data)
-    );
-  }, [schemas.input, inputData.data]);
-
-  function onSubmit() {
+  const submitRef = useRef<(() => void) | null>(null);
+  submitRef.current = () => {
     if (
-      !stopStream &&
-      (!!inputData.errors?.length || !!configData.errors?.length)
+      !props.stopStream &&
+      (!!inputData.errors?.length || !!props.configData.errors?.length)
     ) {
       return;
     }
 
-    if (stopStream) {
-      stopStream();
+    if (props.stopStream) {
+      props.stopStream();
     } else {
-      startStream(inputData.data, configData.data);
+      props.startStream(inputData.data, props.configData.data);
     }
-  }
-
-  const submitRef = useRef<(() => void) | null>(null);
-  submitRef.current = onSubmit;
+  };
 
   useEffect(() => {
     window.addEventListener("keydown", (e) => {
@@ -264,207 +61,162 @@ function App() {
   }, []);
 
   const isSendDisabled =
-    !stopStream && (!!inputData.errors?.length || !!configData.errors?.length);
-
-  if (!schemas.config || !schemas.input) {
-    return <></>;
-  }
+    !props.stopStream &&
+    (!!inputData.errors?.length || !!props.configData.errors?.length);
 
   return (
-    <AppCallbackContext.Provider value={callbacks}>
-      <div className="flex items-center flex-col text-ls-black bg-gradient-to-b from-[#F9FAFB] to-[#EFF8FF] min-h-[100dvh] dark:from-[#0C111C] dark:to-[#0C111C]">
-        <div className="flex flex-col flex-grow gap-4 px-4 pt-6 max-w-[800px] w-full">
-          <h1 className="text-2xl text-left">
-            <strong>🦜 LangServe</strong> Playground
-          </h1>
+    <>
+      <SectionInputs
+        input={props.inputSchema.schema}
+        value={inputData}
+        onChange={(input) => setInputData(input)}
+      />
 
-          {Object.keys(schemas.config).length > 0 && (
-            <div className="flex flex-col gap-3 [&:has(.content>.vertical-layout:first-child:last-child:empty)]:hidden">
-              {!isIframe && (
-                <h2 className="text-xl font-semibold">Configure</h2>
-              )}
+      {props.children}
 
-              <div className="content flex flex-col gap-3">
-                <JsonForms
-                  schema={schemas.config}
-                  data={configData.data}
-                  renderers={renderers}
-                  cells={cells}
-                  onChange={({ data, errors }) =>
-                    data
-                      ? setConfigData({ data, errors, defaults: false })
-                      : undefined
-                  }
-                />
+      <div className="flex-grow md:hidden" />
 
-                {!!configData.errors?.length && configData.data && (
-                  <div className="bg-background rounded-xl">
-                    <div className="bg-red-500/10 text-red-700 dark:text-red-300 rounded-xl p-3">
-                      <strong className="font-bold">Validation Errors</strong>
-                      <ul className="list-disc pl-5">
-                        {configData.errors?.map((e, i) => (
-                          <li key={i}>{e.message}</li>
-                        ))}
-                      </ul>
-                    </div>
-                  </div>
-                )}
-              </div>
+      <div className="gap-4 grid grid-cols-2 sticky -mx-4 px-4 py-4 bottom-0 bg-background md:static md:bg-transparent">
+        <div className="md:hidden absolute inset-x-0 bottom-full h-5 bg-gradient-to-t from-black/5 to-black/0" />
+
+        <ShareDialog config={props.configData.data}>
+          <button
+            type="button"
+            className="px-4 py-3 gap-3 font-medium border border-divider-700 rounded-full flex items-center justify-center hover:bg-divider-500/50 active:bg-divider-500 transition-colors"
+          >
+            <ShareIcon className="flex-shrink-0" /> <span>Share</span>
+          </button>
+        </ShareDialog>
+
+        <SubmitButton
+          disabled={isSendDisabled}
+          onSubmit={submitRef.current}
+          isLoading={!!props.stopStream}
+        />
+      </div>
+    </>
+  );
+}
+
+function ConfigPlayground(props: {
+  configSchema: {
+    schema: JsonSchema;
+    defaults: unknown;
+  };
+}) {
+  const urlState = getStateFromUrl(window.location.href);
+  const [configData, setConfigData] = useState<ConfigValue>({
+    data: urlState.configFromUrl ?? props.configSchema.defaults,
+    errors: [],
+    defaults: true,
+  });
+
+  const feedback = useFeedback();
+
+  // input schema is derived from config data
+  const [debouncedConfigData, debounceState] = useDebounce(
+    configData.data,
+    500
+  );
+
+  const inputSchema = useInputSchema(
+    debouncedConfigData !== props.configSchema.defaults
+      ? debouncedConfigData
+      : undefined
+  );
+
+  const { context, callbacks } = useAppStreamCallbacks();
+  const { startStream, stopStream, latest } = useStreamLog(callbacks);
+
+  return (
+    <AppCallbackContext.Provider value={context}>
+      <SectionConfigure
+        config={props.configSchema.schema}
+        value={configData}
+        onChange={setConfigData}
+      />
+
+      <div
+        className={cn(
+          "flex flex-col flex-grow gap-4 w-full transition-opacity",
+          (inputSchema.isLoading || debounceState.isPending()) &&
+            "opacity-50 pointer-events-none"
+        )}
+      >
+        {inputSchema.error != null ? (
+          <div className="bg-background rounded-xl">
+            <div className="bg-red-500/10 text-red-700 dark:text-red-300 rounded-xl p-3">
+              {inputSchema.error.toString()}
             </div>
-          )}
-
-          {!isIframe && (
-            <div className="flex flex-col gap-3">
-              <h2 className="text-xl font-semibold">Try it</h2>
-
-              <div className="p-4 border border-divider-700 flex flex-col gap-3 rounded-2xl bg-background">
-                <div className="flex items-center justify-between">
-                  <h3 className="font-medium">Inputs</h3>
-                  {isInputResetable && (
-                    <button
-                      type="button"
-                      className="text-sm px-1 -mr-1 py-0.5 rounded-md hover:bg-divider-500/50 active:bg-divider-500 text-ls-gray-100"
-                      onClick={() =>
-                        setInputData({
-                          data: defaults(schemas.input),
-                          errors: [],
-                        })
-                      }
-                    >
-                      Reset
-                    </button>
-                  )}
-                </div>
-
-                <JsonForms
-                  schema={schemas.input}
-                  data={inputData.data}
-                  renderers={renderers}
-                  cells={cells}
-                  onChange={({ data, errors }) =>
-                    setInputData({ data, errors })
-                  }
-                />
-                {!!inputData.errors?.length && inputData.data && (
-                  <div className="bg-red-500/10 text-red-700 dark:text-red-300 rounded-xl p-3">
-                    <strong className="font-bold">Validation Errors</strong>
-                    <ul className="list-disc pl-5">
-                      {inputData.errors?.map((e, i) => (
-                        <li key={i}>{e.message}</li>
-                      ))}
-                    </ul>
-                  </div>
-                )}
-              </div>
-
-              {latest && (
-                <div className="flex flex-col gap-3">
-                  <h2 className="text-xl font-semibold">Output</h2>
-                  <div className="p-4 border border-divider-700 flex flex-col gap-3 rounded-2xl bg-background text-lg whitespace-pre-wrap break-words">
-                    <StreamOutput streamed={latest.streamed_output} />
-                  </div>
-                  <IntermediateSteps latest={latest} />
-                </div>
-              )}
-            </div>
-          )}
-
-          <div className="flex-grow md:hidden" />
-
-          <div className="gap-4 grid grid-cols-2 sticky -mx-4 px-4 py-4 bottom-0 bg-background md:static md:bg-transparent">
-            <div className="md:hidden absolute inset-x-0 bottom-full h-5 bg-gradient-to-t from-black/5 to-black/0" />
-
-            {isIframe ? (
-              <>
-                <button
-                  type="button"
-                  className="px-4 py-3 gap-3 font-medium border border-divider-700 rounded-full flex items-center justify-center hover:bg-divider-500/50 active:bg-divider-500 transition-colors"
-                  onClick={() =>
-                    window.parent?.postMessage({ type: "close" }, "*")
-                  }
-                >
-                  Cancel
-                </button>
-                <button
-                  type="button"
-                  className="px-4 py-3 gap-3 font-medium border border-transparent rounded-full flex items-center justify-center bg-blue-500 hover:bg-blue-600 active:bg-blue-700 disabled:opacity-50 transition-colors"
-                  onClick={() => {
-                    const hash = compressToEncodedURIComponent(
-                      JSON.stringify(configData.data)
-                    );
-
-                    const state = getStateFromUrl(window.location.href);
-                    const targetUrl = `${state.basePath}/c/${hash}`;
-                    window.parent?.postMessage(
-                      {
-                        type: "apply",
-                        value: { targetUrl, config: configData.data },
-                      },
-                      "*"
-                    );
-                  }}
-                >
-                  <span className="text-white">Apply</span>
-                </button>
-              </>
-            ) : (
-              <>
-                <ShareDialog config={configData.data}>
-                  <button
-                    type="button"
-                    className="px-4 py-3 gap-3 font-medium border border-divider-700 rounded-full flex items-center justify-center hover:bg-divider-500/50 active:bg-divider-500 transition-colors"
-                  >
-                    <ShareIcon className="flex-shrink-0" /> <span>Share</span>
-                  </button>
-                </ShareDialog>
-                <button
-                  type="button"
-                  className={cn(
-                    "px-4 py-3 gap-3 font-medium border border-transparent rounded-full flex items-center justify-center bg-blue-500 disabled:opacity-50 transition-colors",
-                    !isSendDisabled
-                      ? "hover:bg-blue-600 active:bg-blue-700"
-                      : ""
-                  )}
-                  onClick={onSubmit}
-                  disabled={isSendDisabled}
-                >
-                  {stopStream ? (
-                    <>
-                      <div role="status">
-                        <svg
-                          aria-hidden="true"
-                          className="w-5 h-5 animate-spin text-white fill-ls-blue"
-                          viewBox="0 0 100 101"
-                          fill="none"
-                          xmlns="http://www.w3.org/2000/svg"
-                        >
-                          <path
-                            d="M100 50.5908C100 78.2051 77.6142 100.591 50 100.591C22.3858 100.591 0 78.2051 0 50.5908C0 22.9766 22.3858 0.59082 50 0.59082C77.6142 0.59082 100 22.9766 100 50.5908ZM9.08144 50.5908C9.08144 73.1895 27.4013 91.5094 50 91.5094C72.5987 91.5094 90.9186 73.1895 90.9186 50.5908C90.9186 27.9921 72.5987 9.67226 50 9.67226C27.4013 9.67226 9.08144 27.9921 9.08144 50.5908Z"
-                            fill="currentColor"
-                          />
-                          <path
-                            d="M93.9676 39.0409C96.393 38.4038 97.8624 35.9116 97.0079 33.5539C95.2932 28.8227 92.871 24.3692 89.8167 20.348C85.8452 15.1192 80.8826 10.7238 75.2124 7.41289C69.5422 4.10194 63.2754 1.94025 56.7698 1.05124C51.7666 0.367541 46.6976 0.446843 41.7345 1.27873C39.2613 1.69328 37.813 4.19778 38.4501 6.62326C39.0873 9.04874 41.5694 10.4717 44.0505 10.1071C47.8511 9.54855 51.7191 9.52689 55.5402 10.0491C60.8642 10.7766 65.9928 12.5457 70.6331 15.2552C75.2735 17.9648 79.3347 21.5619 82.5849 25.841C84.9175 28.9121 86.7997 32.2913 88.1811 35.8758C89.083 38.2158 91.5421 39.6781 93.9676 39.0409Z"
-                            fill="currentFill"
-                          />
-                        </svg>
-                        <span className="sr-only">Loading...</span>
-                      </div>
-                      <span className="text-white">Stop</span>
-                    </>
-                  ) : (
-                    <>
-                      <SendIcon className="flex-shrink-0" />
-                      <span className="text-white">Start</span>
-                    </>
-                  )}
-                </button>
-              </>
-            )}
           </div>
-        </div>
+        ) : (
+          <>
+            {inputSchema.data != null ? (
+              <InputPlayground
+                configSchema={props.configSchema}
+                inputSchema={inputSchema.data}
+                configData={configData}
+                startStream={startStream}
+                stopStream={stopStream}
+              >
+                {latest && (
+                  <div className="flex flex-col gap-3">
+                    <h2 className="text-xl font-semibold">Output</h2>
+                    <div className="p-4 border border-divider-700 flex flex-col gap-3 rounded-2xl bg-background text-lg whitespace-pre-wrap break-words relative group">
+                      <StreamOutput streamed={latest.streamed_output} />
+
+                      {feedback.data && latest.id ? (
+                        <div className="absolute right-4 top-4 flex items-center gap-2 transition-opacity opacity-0 focus-within:opacity-100 group-hover:opacity-100">
+                          <CorrectnessFeedback
+                            key={latest.id}
+                            runId={latest.id}
+                          />
+                        </div>
+                      ) : null}
+                    </div>
+                    <IntermediateSteps
+                      latest={latest}
+                      feedbackEnabled={!!feedback.data}
+                    />
+                  </div>
+                )}
+              </InputPlayground>
+            ) : null}
+          </>
+        )}
       </div>
     </AppCallbackContext.Provider>
   );
 }
 
+function Playground() {
+  const configSchema = useConfigSchema();
+  if (configSchema.isLoading) return null;
+
+  if (configSchema.error != null) {
+    return (
+      <div className="bg-background rounded-xl">
+        <div className="bg-red-500/10 text-red-700 dark:text-red-300 rounded-xl p-3">
+          {configSchema.error.toString()}
+        </div>
+      </div>
+    );
+  }
+  if (configSchema.data == null) return "No config schema found";
+  return <ConfigPlayground configSchema={configSchema.data} />;
+}
+
+export function App() {
+  return (
+    <div className="flex items-center flex-col text-ls-black bg-gradient-to-b from-[#F9FAFB] to-[#EFF8FF] min-h-[100dvh] dark:from-[#0C111C] dark:to-[#0C111C]">
+      <div className="flex flex-col flex-grow gap-4 px-4 pt-6 max-w-[800px] w-full">
+        <h1 className="text-2xl text-left">
+          <strong>🦜 LangServe</strong> Playground
+        </h1>
+        <Playground />
+      </div>
+    </div>
+  );
+}
+
 export default App;

langserve/playground/src/assets/CircleSpinIcon.svg

@@ -0,0 +1,15 @@
+<svg
+  aria-hidden="true"
+  viewBox="0 0 100 101"
+  fill="none"
+  xmlns="http://www.w3.org/2000/svg"
+>
+  <path
+    d="M100 50.5908C100 78.2051 77.6142 100.591 50 100.591C22.3858 100.591 0 78.2051 0 50.5908C0 22.9766 22.3858 0.59082 50 0.59082C77.6142 0.59082 100 22.9766 100 50.5908ZM9.08144 50.5908C9.08144 73.1895 27.4013 91.5094 50 91.5094C72.5987 91.5094 90.9186 73.1895 90.9186 50.5908C90.9186 27.9921 72.5987 9.67226 50 9.67226C27.4013 9.67226 9.08144 27.9921 9.08144 50.5908Z"
+    fill="currentColor"
+  />
+  <path
+    d="M93.9676 39.0409C96.393 38.4038 97.8624 35.9116 97.0079 33.5539C95.2932 28.8227 92.871 24.3692 89.8167 20.348C85.8452 15.1192 80.8826 10.7238 75.2124 7.41289C69.5422 4.10194 63.2754 1.94025 56.7698 1.05124C51.7666 0.367541 46.6976 0.446843 41.7345 1.27873C39.2613 1.69328 37.813 4.19778 38.4501 6.62326C39.0873 9.04874 41.5694 10.4717 44.0505 10.1071C47.8511 9.54855 51.7191 9.52689 55.5402 10.0491C60.8642 10.7766 65.9928 12.5457 70.6331 15.2552C75.2735 17.9648 79.3347 21.5619 82.5849 25.841C84.9175 28.9121 86.7997 32.2913 88.1811 35.8758C89.083 38.2158 91.5421 39.6781 93.9676 39.0409Z"
+    fill="currentFill"
+  />
+</svg>

langserve/playground/src/assets/ThumbsDownIcon.svg

@@ -0,0 +1,6 @@
+<svg width="24" height="24" viewBox="0 0 24 24" fill="none" xmlns="http://www.w3.org/2000/svg">
+  <path
+    d="M17 2V13M22 9.8V5.2C22 4.07989 22 3.51984 21.782 3.09202C21.5903 2.71569 21.2843 2.40973 20.908 2.21799C20.4802 2 19.9201 2 18.8 2H8.11802C6.65654 2 5.92579 2 5.33559 2.26743C4.81541 2.50314 4.37331 2.88242 4.06125 3.36072C3.70718 3.90339 3.59606 4.62564 3.37383 6.07012L2.85076 9.47012C2.55765 11.3753 2.4111 12.3279 2.69381 13.0691C2.94195 13.7197 3.40866 14.2637 4.01393 14.6079C4.70354 15 5.66734 15 7.59494 15H8.40001C8.96006 15 9.24009 15 9.454 15.109C9.64216 15.2049 9.79514 15.3578 9.89101 15.546C10 15.7599 10 16.0399 10 16.6V19.5342C10 20.896 11.104 22 12.4658 22C12.7907 22 13.085 21.8087 13.217 21.5119L16.5777 13.9502C16.7306 13.6062 16.807 13.4343 16.9278 13.3082C17.0346 13.1967 17.1657 13.1115 17.311 13.0592C17.4753 13 17.6634 13 18.0398 13H18.8C19.9201 13 20.4802 13 20.908 12.782C21.2843 12.5903 21.5903 12.2843 21.782 11.908C22 11.4802 22 10.9201 22 9.8Z"
+    stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" />
+</svg>
+  

langserve/playground/src/assets/ThumbsUpIcon.svg

@@ -0,0 +1,6 @@
+<svg width="24" height="24" viewBox="0 0 24 24" fill="none" xmlns="http://www.w3.org/2000/svg">
+  <path
+    d="M7 22V11M2 13V20C2 21.1046 2.89543 22 4 22H17.4262C18.907 22 20.1662 20.9197 20.3914 19.4562L21.4683 12.4562C21.7479 10.6389 20.3418 9 18.5032 9H15C14.4477 9 14 8.55228 14 8V4.46584C14 3.10399 12.896 2 11.5342 2C11.2093 2 10.915 2.1913 10.7831 2.48812L7.26394 10.4061C7.10344 10.7673 6.74532 11 6.35013 11H4C2.89543 11 2 11.8954 2 13Z"
+    stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" />
+</svg>
+  

langserve/playground/src/components/ChatMessageTuplesControlRenderer.tsx

@@ -11,7 +11,8 @@ import {
 import { AutosizeTextarea } from "./AutosizeTextarea";
 import { isJsonSchemaExtra } from "../utils/schema";
 import { useStreamCallback } from "../useStreamCallback";
-import { traverseNaiveJsonPath } from "../utils/path";
+import { getNormalizedJsonPath, traverseNaiveJsonPath } from "../utils/path";
+import { getMessageContent } from "../utils/messages";
 
 type MessageTuple = [string, string];
 
@@ -53,9 +54,25 @@ export const ChatMessageTuplesControlRenderer = withJsonFormsControlProps(
       const widget = props.schema.extra.widget;
       if (!("input" in widget) && !("output" in widget)) return;
 
-      const human = traverseNaiveJsonPath(ctx.input, widget.input ?? "");
-      const ai = traverseNaiveJsonPath(ctx.output, widget.output ?? "");
-      props.handleChange(props.path, [...data, [human, ai]]);
+      const inputPath = getNormalizedJsonPath(widget.input ?? "");
+      const outputPath = getNormalizedJsonPath(widget.output ?? "");
+
+      const isSingleOutputKey =
+        ctx.output != null &&
+        Object.keys(ctx.output).length === 1 &&
+        Object.keys(ctx.output)[0] === "output";
+
+      const human = traverseNaiveJsonPath(ctx.input, inputPath);
+      let ai = traverseNaiveJsonPath(ctx.output, outputPath);
+
+      if (isSingleOutputKey) {
+        ai = traverseNaiveJsonPath(ai, ["output", ...outputPath]) ?? ai;
+      }
+
+      ai = getMessageContent(ai);
+      if (typeof human === "string" && typeof ai === "string") {
+        props.handleChange(props.path, [...data, [human, ai]]);
+      }
     });
 
     return (

langserve/playground/src/components/ChatMessagesControlRenderer.tsx

@@ -12,7 +12,7 @@ import {
 } from "@jsonforms/core";
 import { AutosizeTextarea } from "./AutosizeTextarea";
 import { useStreamCallback } from "../useStreamCallback";
-import { traverseNaiveJsonPath } from "../utils/path";
+import { getNormalizedJsonPath, traverseNaiveJsonPath } from "../utils/path";
 import { isJsonSchemaExtra } from "../utils/schema";
 import * as ToggleGroup from "@radix-ui/react-toggle-group";
 
@@ -122,8 +122,20 @@ export const ChatMessagesControlRenderer = withJsonFormsControlProps(
       const widget = props.schema.extra.widget;
       if (!("input" in widget) && !("output" in widget)) return;
 
-      const human = traverseNaiveJsonPath(ctx.input, widget.input ?? "");
-      const ai = traverseNaiveJsonPath(ctx.output, widget.output ?? "");
+      const inputPath = getNormalizedJsonPath(widget.input ?? "");
+      const outputPath = getNormalizedJsonPath(widget.output ?? "");
+
+      const human = traverseNaiveJsonPath(ctx.input, inputPath);
+      let ai = traverseNaiveJsonPath(ctx.output, outputPath);
+
+      const isSingleOutputKey =
+        ctx.output != null &&
+        Object.keys(ctx.output).length === 1 &&
+        Object.keys(ctx.output)[0] === "output";
+
+      if (isSingleOutputKey) {
+        ai = traverseNaiveJsonPath(ai, ["output", ...outputPath]) ?? ai;
+      }
 
       const humanMsg = constructMessage(human, "human");
       const aiMsg = constructMessage(ai, "ai");
@@ -296,12 +308,6 @@ export const ChatMessagesControlRenderer = withJsonFormsControlProps(
                         message.additional_kwargs?.function_call.name ?? ""
                       }
                       onChange={(e) => {
-                        console.log(
-                          Paths.compose(
-                            msgPath,
-                            "additional_kwargs.function_call.name"
-                          )
-                        );
                         props.handleChange(
                           Paths.compose(
                             msgPath,

langserve/playground/src/components/CustomAnyOfRenderer.tsx

@@ -5,7 +5,7 @@ import {
   JsonSchema,
   isAnyOfControl,
 } from "@jsonforms/core";
-import { renderers, cells } from "../App";
+import { renderers, cells } from "../renderers";
 
 export const CustomAnyOfRenderer = withJsonFormsAnyOfProps((props) => {
   const anyOfRenderInfos = createCombinatorRenderInfos(

langserve/playground/src/components/IntermediateSteps.tsx

@@ -4,8 +4,12 @@ import ChevronRight from "../assets/ChevronRight.svg?react";
 import { RunState } from "../useStreamLog";
 import { cn } from "../utils/cn";
 import { str } from "../utils/str";
+import { CorrectnessFeedback } from "./feedback/CorrectnessFeedback";
 
-export function IntermediateSteps(props: { latest: RunState }) {
+export function IntermediateSteps(props: {
+  latest: RunState;
+  feedbackEnabled: boolean;
+}) {
   const [expanded, setExpanded] = useState(false);
   const length = Object.values(props.latest.logs).length;
   const disabled = length === 0;
@@ -41,9 +45,16 @@ export function IntermediateSteps(props: { latest: RunState }) {
                 <strong className="text-sm font-medium">{log.name}</strong>
                 <p className="text-sm">{dayjs.utc(log.start_time).fromNow()}</p>
               </div>
-              <pre className="break-words whitespace-pre-wrap min-w-0 text-sm bg-ls-gray-400 rounded-lg p-3">
-                {str(log.final_output) ?? "..."}
-              </pre>
+              <div className="bg-ls-gray-400 rounded-lg p-3 relative group">
+                <pre className="break-words whitespace-pre-wrap min-w-0 text-sm">
+                  {str(log.final_output) ?? "..."}
+                </pre>
+                {props.feedbackEnabled && log.id ? (
+                  <div className="absolute right-3 top-3 flex items-center gap-2 transition-opacity opacity-0 focus-within:opacity-100 group-hover:opacity-100">
+                    <CorrectnessFeedback key={log.id} runId={log.id} />
+                  </div>
+                ) : null}
+              </div>
             </div>
           ))}
         </div>

langserve/playground/src/components/ShareDialog.tsx

@@ -4,37 +4,11 @@ import CodeIcon from "../assets/CodeIcon.svg?react";
 import PadlockIcon from "../assets/PadlockIcon.svg?react";
 import CopyIcon from "../assets/CopyIcon.svg?react";
 import CheckCircleIcon from "../assets/CheckCircleIcon.svg?react";
-import {
-  compressToEncodedURIComponent,
-  decompressFromEncodedURIComponent,
-} from "lz-string";
+import { compressToEncodedURIComponent } from "lz-string";
+import { getStateFromUrl } from "../utils/url";
 
 const URL_LENGTH_LIMIT = 2000;
 
-export function getStateFromUrl(path: string) {
-  let configFromUrl = null;
-  let basePath = path;
-  if (basePath.endsWith("/")) {
-    basePath = basePath.slice(0, -1);
-  }
-
-  if (basePath.endsWith("/playground")) {
-    basePath = basePath.slice(0, -"/playground".length);
-  }
-
-  // check if we can omit the last segment
-  const [configHash, c, ...rest] = basePath.split("/").reverse();
-  if (c === "c") {
-    basePath = rest.reverse().join("/");
-    try {
-      configFromUrl = JSON.parse(decompressFromEncodedURIComponent(configHash));
-    } catch (error) {
-      console.error(error);
-    }
-  }
-  return { basePath, configFromUrl };
-}
-
 function CopyButton(props: { value: string }) {
   const [copied, setCopied] = useState(false);
   const cbRef = useRef<number | null>(null);
@@ -121,7 +95,6 @@ const result = await chain.invoke({ ... });
                     <PadlockIcon className="mx-3" />
                     <div className="overflow-auto whitespace-nowrap py-3 no-scrollbar text-ls-gray-100">
                       {playgroundUrl.split("://")[1]}
-                      PadlockIcon
                     </div>
                     <CopyButton value={playgroundUrl} />
                   </div>

langserve/playground/src/components/SubmitButton.tsx

@@ -0,0 +1,50 @@
+import SendIcon from "../assets/SendIcon.svg?react";
+import { cn } from "../utils/cn";
+
+export function SubmitButton(props: {
+  disabled: boolean;
+  isLoading?: boolean;
+  onSubmit: () => void;
+}) {
+  return (
+    <button
+      type="button"
+      className={cn(
+        "px-4 py-3 gap-3 font-medium border border-transparent rounded-full flex items-center justify-center bg-blue-500 disabled:opacity-50 transition-colors",
+        !props.disabled ? "hover:bg-blue-600 active:bg-blue-700" : ""
+      )}
+      onClick={props.onSubmit}
+      disabled={props.disabled}
+    >
+      {props.isLoading ? (
+        <>
+          <div role="status">
+            <svg
+              aria-hidden="true"
+              className="w-5 h-5 animate-spin text-white fill-ls-blue"
+              viewBox="0 0 100 101"
+              fill="none"
+              xmlns="http://www.w3.org/2000/svg"
+            >
+              <path
+                d="M100 50.5908C100 78.2051 77.6142 100.591 50 100.591C22.3858 100.591 0 78.2051 0 50.5908C0 22.9766 22.3858 0.59082 50 0.59082C77.6142 0.59082 100 22.9766 100 50.5908ZM9.08144 50.5908C9.08144 73.1895 27.4013 91.5094 50 91.5094C72.5987 91.5094 90.9186 73.1895 90.9186 50.5908C90.9186 27.9921 72.5987 9.67226 50 9.67226C27.4013 9.67226 9.08144 27.9921 9.08144 50.5908Z"
+                fill="currentColor"
+              />
+              <path
+                d="M93.9676 39.0409C96.393 38.4038 97.8624 35.9116 97.0079 33.5539C95.2932 28.8227 92.871 24.3692 89.8167 20.348C85.8452 15.1192 80.8826 10.7238 75.2124 7.41289C69.5422 4.10194 63.2754 1.94025 56.7698 1.05124C51.7666 0.367541 46.6976 0.446843 41.7345 1.27873C39.2613 1.69328 37.813 4.19778 38.4501 6.62326C39.0873 9.04874 41.5694 10.4717 44.0505 10.1071C47.8511 9.54855 51.7191 9.52689 55.5402 10.0491C60.8642 10.7766 65.9928 12.5457 70.6331 15.2552C75.2735 17.9648 79.3347 21.5619 82.5849 25.841C84.9175 28.9121 86.7997 32.2913 88.1811 35.8758C89.083 38.2158 91.5421 39.6781 93.9676 39.0409Z"
+                fill="currentFill"
+              />
+            </svg>
+            <span className="sr-only">Loading...</span>
+          </div>
+          <span className="text-white">Stop</span>
+        </>
+      ) : (
+        <>
+          <SendIcon className="flex-shrink-0" />
+          <span className="text-white">Start</span>
+        </>
+      )}
+    </button>
+  );
+}

langserve/playground/src/components/feedback/CorrectnessFeedback.tsx

@@ -0,0 +1,89 @@
+import ThumbsUpIcon from "../../assets/ThumbsUpIcon.svg?react";
+import ThumbsDownIcon from "../../assets/ThumbsDownIcon.svg?react";
+import CircleSpinIcon from "../../assets/CircleSpinIcon.svg?react";
+import { resolveApiUrl } from "../../utils/url";
+import { useState } from "react";
+import { cn } from "../../utils/cn";
+import useSWRMutation from "swr/mutation";
+
+const useFeedbackMutation = (runId: string) => {
+  interface FeedbackArguments {
+    key: string;
+    score: number;
+  }
+
+  const [lastArg, setLastArg] = useState<FeedbackArguments | null>(null);
+
+  const mutation = useSWRMutation(
+    ["feedback", runId],
+    async ([, runId], { arg }: { arg: FeedbackArguments }) => {
+      const payload = { run_id: runId, key: arg.key, score: arg.score };
+      setLastArg(arg);
+
+      const request = await fetch(resolveApiUrl("/feedback"), {
+        method: "POST",
+        headers: { "Content-Type": "application/json" },
+        body: JSON.stringify(payload),
+      });
+
+      if (!request.ok) throw new Error(`Failed request ${request.status}`);
+      const json: {
+        id: string;
+        score: number;
+      } = await request.json();
+
+      return json;
+    }
+  );
+
+  return { lastArg: mutation.isMutating ? lastArg : null, mutation };
+};
+
+export function CorrectnessFeedback(props: { runId: string }) {
+  const score = useFeedbackMutation(props.runId);
+
+  if (props.runId == null) return null;
+  return (
+    <>
+      <button
+        type="button"
+        className={cn(
+          "border focus-within:border-ls-blue focus-within:outline-none bg-background rounded p-1 border-divider-700 hover:bg-divider-500/50 active:bg-divider-500",
+          score.mutation.data?.score === 1 && "text-teal-500"
+        )}
+        disabled={score.mutation.isMutating}
+        onClick={() => {
+          if (score.mutation.data?.score !== 1) {
+            score.mutation.trigger({ key: "correctness", score: 1 });
+          }
+        }}
+      >
+        {score.lastArg?.score === 1 ? (
+          <CircleSpinIcon className="animate-spin w-4 h-4 text-white/50 fill-white" />
+        ) : (
+          <ThumbsUpIcon className="w-4 h-4" />
+        )}
+      </button>
+
+      <button
+        type="button"
+        className={cn(
+          "border focus-within:border-ls-blue focus-within:outline-none bg-background rounded p-1 border-divider-700 hover:bg-divider-500/50 active:bg-divider-500",
+          score.mutation.data?.score === 0 && "text-red-500"
+        )}
+        disabled={score.mutation.isMutating}
+        onClick={() => {
+          if (score.mutation.data?.score !== 0) {
+            score.mutation.trigger({ key: "correctness", score: 0 });
+          }
+        }}
+      >
+        {score.lastArg?.score === 0 ? (
+          <CircleSpinIcon className="animate-spin w-4 h-4 text-white/50 fill-white" />
+        ) : (
+          <ThumbsDownIcon className="w-4 h-4" />
+        )}
+      </button>
+    </>
+  );
+}

langserve/playground/src/main.tsx

@@ -1,4 +1,11 @@
 import ReactDOM from "react-dom/client";
 import App from "./App.tsx";
 
+import dayjs from "dayjs";
+import utc from "dayjs/plugin/utc";
+import relativeDate from "dayjs/plugin/relativeTime";
+
+dayjs.extend(relativeDate);
+dayjs.extend(utc);
+
 ReactDOM.createRoot(document.getElementById("root")!).render(<App />);

langserve/playground/src/renderers.tsx

@@ -0,0 +1,111 @@
+import {
+  materialAllOfControlTester,
+  MaterialAllOfRenderer,
+  MaterialObjectRenderer,
+  materialOneOfControlTester,
+  MaterialOneOfRenderer,
+} from "@jsonforms/material-renderers";
+import {
+  BooleanCell,
+  DateCell,
+  DateTimeCell,
+  EnumCell,
+  IntegerCell,
+  NumberCell,
+  SliderCell,
+  TimeCell,
+  booleanCellTester,
+  dateCellTester,
+  dateTimeCellTester,
+  enumCellTester,
+  integerCellTester,
+  numberCellTester,
+  sliderCellTester,
+  textAreaCellTester,
+  textCellTester,
+  timeCellTester,
+  vanillaRenderers,
+  InputControl,
+} from "@jsonforms/vanilla-renderers";
+import {
+  RankedTester,
+  rankWith,
+  and,
+  uiTypeIs,
+  schemaMatches,
+  schemaTypeIs,
+} from "@jsonforms/core";
+import CustomArrayControlRenderer, {
+  materialArrayControlTester,
+} from "./components/CustomArrayControlRenderer";
+import CustomTextAreaCell from "./components/CustomTextAreaCell";
+import JsonTextAreaCell from "./components/JsonTextAreaCell";
+import {
+  chatMessagesTester,
+  ChatMessagesControlRenderer,
+} from "./components/ChatMessagesControlRenderer";
+import {
+  ChatMessageTuplesControlRenderer,
+  chatMessagesTupleTester,
+} from "./components/ChatMessageTuplesControlRenderer";
+import {
+  fileBase64Tester,
+  FileBase64ControlRenderer,
+} from "./components/FileBase64Tester";
+import {
+  customAnyOfTester,
+  CustomAnyOfRenderer,
+} from "./components/CustomAnyOfRenderer";
+
+const isObjectWithPropertiesControl = rankWith(
+  2,
+  and(
+    uiTypeIs("Control"),
+    schemaTypeIs("object"),
+    schemaMatches((schema) =>
+      Object.prototype.hasOwnProperty.call(schema, "properties")
+    )
+  )
+);
+const isObject = rankWith(1, and(uiTypeIs("Control"), schemaTypeIs("object")));
+const isElse = rankWith(1, and(uiTypeIs("Control")));
+
+export const renderers = [
+  ...vanillaRenderers,
+
+  // use material renderers to handle objects and json schema references
+  // they should yield the rendering to simpler cells
+  { tester: isObjectWithPropertiesControl, renderer: MaterialObjectRenderer },
+  { tester: materialAllOfControlTester, renderer: MaterialAllOfRenderer },
+  { tester: materialOneOfControlTester, renderer: MaterialOneOfRenderer },
+
+  { tester: customAnyOfTester, renderer: CustomAnyOfRenderer },
+
+  // custom renderers
+  { tester: materialArrayControlTester, renderer: CustomArrayControlRenderer },
+  { tester: isObject, renderer: InputControl },
+  { tester: chatMessagesTester, renderer: ChatMessagesControlRenderer },
+  {
+    tester: chatMessagesTupleTester,
+    renderer: ChatMessageTuplesControlRenderer,
+  },
+  { tester: fileBase64Tester, renderer: FileBase64ControlRenderer },
+];
+const nestedArrayControlTester: RankedTester = rankWith(1, (_, jsonSchema) => {
+  return jsonSchema.type === "array";
+});
+
+export const cells = [
+  { tester: booleanCellTester, cell: BooleanCell },
+  { tester: dateCellTester, cell: DateCell },
+  { tester: dateTimeCellTester, cell: DateTimeCell },
+  { tester: enumCellTester, cell: EnumCell },
+  { tester: integerCellTester, cell: IntegerCell },
+  { tester: numberCellTester, cell: NumberCell },
+  { tester: sliderCellTester, cell: SliderCell },
+  { tester: textAreaCellTester, cell: CustomTextAreaCell },
+  { tester: textCellTester, cell: CustomTextAreaCell },
+  { tester: timeCellTester, cell: TimeCell },
+  { tester: nestedArrayControlTester, cell: CustomArrayControlRenderer },
+  { tester: isElse, cell: JsonTextAreaCell },
+];

langserve/playground/src/sections/SectionConfigure.tsx

@@ -0,0 +1,50 @@
+import { JsonForms } from "@jsonforms/react";
+import { JsonFormsCore, JsonSchema } from "@jsonforms/core";
+import { renderers, cells } from "../renderers";
+
+export type ConfigValue = Pick<JsonFormsCore, "data" | "errors"> & {
+  defaults: boolean;
+};
+
+export function SectionConfigure(props: {
+  config: JsonSchema | undefined;
+  value: ConfigValue;
+  onChange: (value: ConfigValue) => void;
+}) {
+  if (props.config == null || Object.keys(props.config).length === 0) {
+    return null;
+  }
+
+  return (
+    <div className="flex flex-col gap-3 [&:has(.content>.vertical-layout:first-child:last-child:empty)]:hidden">
+      <h2 className="text-xl font-semibold">Configure</h2>
+
+      <div className="content flex flex-col gap-3">
+        <JsonForms
+          schema={props.config}
+          data={props.value.data}
+          renderers={renderers}
+          cells={cells}
+          onChange={({ data, errors }) => {
+            if (data) {
+              props.onChange({ data, errors, defaults: false });
+            }
+          }}
+        />
+
+        {!!props.value.errors?.length && props.value.data && (
+          <div className="bg-background rounded-xl">
+            <div className="bg-red-500/10 text-red-700 dark:text-red-300 rounded-xl p-3">
+              <strong className="font-bold">Validation Errors</strong>
+              <ul className="list-disc pl-5">
+                {props.value.errors?.map((e, i) => (
+                  <li key={i}>{e.message}</li>
+                ))}
+              </ul>
+            </div>
+          </div>
+        )}
+      </div>
+    </div>
+  );
+}

langserve/playground/src/sections/SectionInputs.tsx

@@ -0,0 +1,64 @@
+import { useMemo } from "react";
+import defaults from "../utils/defaults";
+import { JsonForms } from "@jsonforms/react";
+import { JsonFormsCore, JsonSchema } from "@jsonforms/core";
+import { renderers, cells } from "../renderers";
+
+export type InputValue = Pick<JsonFormsCore, "data" | "errors">;
+
+export function SectionInputs(props: {
+  input: JsonSchema | undefined;
+  value: InputValue;
+  onChange: (value: InputValue) => void;
+}) {
+  const isInputResetable = useMemo(() => {
+    if (!props.input) return false;
+    return (
+      JSON.stringify(defaults(props.input)) !== JSON.stringify(props.value.data)
+    );
+  }, [props.input, props.value.data]);
+
+  return (
+    <div className="flex flex-col gap-3">
+      <h2 className="text-xl font-semibold">Try it</h2>
+
+      <div className="p-4 border border-divider-700 flex flex-col gap-3 rounded-2xl bg-background">
+        <div className="flex items-center justify-between">
+          <h3 className="font-medium">Inputs</h3>
+          {isInputResetable && (
+            <button
+              type="button"
+              className="text-sm px-1 -mr-1 py-0.5 rounded-md hover:bg-divider-500/50 active:bg-divider-500 text-ls-gray-100"
+              onClick={() =>
+                props.onChange({
+                  data: defaults(props.input),
+                  errors: [],
+                })
+              }
+            >
+              Reset
+            </button>
+          )}
+        </div>
+
+        <JsonForms
+          schema={props.input}
+          data={props.value.data}
+          renderers={renderers}
+          cells={cells}
+          onChange={({ data, errors }) => props.onChange({ data, errors })}
+        />
+        {!!props.value.errors?.length && props.value.data && (
+          <div className="bg-red-500/10 text-red-700 dark:text-red-300 rounded-xl p-3">
+            <strong className="font-bold">Validation Errors</strong>
+            <ul className="list-disc pl-5">
+              {props.value.errors?.map((e, i) => (
+                <li key={i}>{e.message}</li>
+              ))}
+            </ul>
+          </div>
+        )}
+      </div>
+    </div>
+  );
+}

langserve/playground/src/useSchemas.tsx

@@ -1,9 +1,10 @@
-import { useEffect, useState } from "react";
 import { resolveApiUrl } from "./utils/url";
 import { simplifySchema } from "./utils/simplifySchema";
-import { JsonFormsCore } from "@jsonforms/core";
+import { JsonSchema } from "@jsonforms/core";
 import { compressToEncodedURIComponent } from "lz-string";
-import { useDebounce } from "use-debounce";
+
+import useSWR from "swr";
+import defaults from "./utils/defaults";
 
 declare global {
   interface Window {
@@ -11,66 +12,72 @@ declare global {
     CONFIG_SCHEMA?: any;
     // eslint-disable-next-line @typescript-eslint/no-explicit-any
     INPUT_SCHEMA?: any;
+    // eslint-disable-next-line @typescript-eslint/no-explicit-any
+    FEEDBACK_ENABLED?: any;
   }
 }
 
-export function useSchemas(
-  configData: Pick<JsonFormsCore, "data" | "errors"> & { defaults: boolean }
-) {
-  const [schemas, setSchemas] = useState<{
-    // eslint-disable-next-line @typescript-eslint/no-explicit-any
-    config: null | any;
-    // eslint-disable-next-line @typescript-eslint/no-explicit-any
-    input: null | any;
-  }>({
-    config: null,
-    input: null,
+export function useFeedback() {
+  return useSWR(["/feedback"], async () => {
+    if (!import.meta.env.DEV && window.FEEDBACK_ENABLED) {
+      return window.FEEDBACK_ENABLED === "true";
+    }
+
+    const response = await fetch(resolveApiUrl("/feedback"), {
+      method: "HEAD",
+    });
+    return response.ok;
   });
-
-  useEffect(() => {
-    async function save() {
-      if (import.meta.env.DEV) {
-        const [config, input] = await Promise.all([
-          fetch(resolveApiUrl("/config_schema"))
-            .then((r) => r.json())
-            .then(simplifySchema),
-          fetch(resolveApiUrl("/input_schema"))
-            .then((r) => r.json())
-            .then(simplifySchema),
-        ]);
-        setSchemas({ config, input });
-      } else {
-        setSchemas({
-          config: window.CONFIG_SCHEMA
-            ? await simplifySchema(window.CONFIG_SCHEMA)
-            : null,
-          input: window.INPUT_SCHEMA
-            ? await simplifySchema(window.INPUT_SCHEMA)
-            : null,
-        });
-      }
-    }
-
-    save();
-  }, []);
-
-  const [debouncedConfigData] = useDebounce(configData, 500);
-
-  useEffect(() => {
-    if (!debouncedConfigData.defaults) {
-      fetch(
-        resolveApiUrl(
-          `/c/${compressToEncodedURIComponent(
-            JSON.stringify(debouncedConfigData.data)
-          )}/input_schema`
-        )
-      )
-        .then((r) => r.json())
-        .then(simplifySchema)
-        .then((input) => setSchemas((current) => ({ ...current, input })))
-        .catch(() => {}); // ignore errors, eg. due to incomplete config
-    }
-  }, [debouncedConfigData]);
-
-  return schemas;
+}
+
+export function useConfigSchema() {
+  return useSWR(["/config_schema"], async () => {
+    let schema: JsonSchema | null = null;
+    if (!import.meta.env.DEV && window.CONFIG_SCHEMA) {
+      schema = await simplifySchema(window.CONFIG_SCHEMA);
+    } else {
+      const response = await fetch(resolveApiUrl(`/config_schema`));
+      if (!response.ok) throw new Error(await response.text());
+
+      const json = await response.json();
+      schema = await simplifySchema(json);
+    }
+
+    if (schema == null) return null;
+    return {
+      schema,
+      defaults: defaults(schema),
+    };
+  });
+}
+
+export function useInputSchema(configData?: unknown) {
+  return useSWR(
+    ["/input_schema", configData],
+    async ([, configData]) => {
+      // TODO: this won't work if we're already seeing a prefixed URL
+      const prefix = configData
+        ? `/c/${compressToEncodedURIComponent(JSON.stringify(configData))}`
+        : "";
+
+      let schema: JsonSchema | null = null;
+
+      if (!prefix && !import.meta.env.DEV && window.INPUT_SCHEMA) {
+        schema = await simplifySchema(window.INPUT_SCHEMA);
+      } else {
+        const response = await fetch(resolveApiUrl(`${prefix}/input_schema`));
+        if (!response.ok) throw new Error(await response.text());
+
+        const json = await response.json();
+        schema = await simplifySchema(json);
+      }
+
+      if (schema == null) return null;
+      return {
+        schema,
+        defaults: defaults(schema),
+      };
+    },
+    { keepPreviousData: true }
+  );
 }

langserve/playground/src/useStreamCallback.tsx

@@ -14,6 +14,35 @@ export const AppCallbackContext = createContext<MutableRefObject<{
   onError: Exclude<StreamCallback["onError"], undefined>[];
 }> | null>(null);
 
+export function useAppStreamCallbacks() {
+  // callbacks handling
+  const context = useRef<{
+    onStart: Exclude<StreamCallback["onStart"], undefined>[];
+    onSuccess: Exclude<StreamCallback["onSuccess"], undefined>[];
+    onError: Exclude<StreamCallback["onError"], undefined>[];
+  }>({ onStart: [], onSuccess: [], onError: [] });
+
+  const callbacks: StreamCallback = {
+    onStart(...args) {
+      for (const callback of context.current.onStart) {
+        callback(...args);
+      }
+    },
+    onSuccess(...args) {
+      for (const callback of context.current.onSuccess) {
+        callback(...args);
+      }
+    },
+    onError(...args) {
+      for (const callback of context.current.onError) {
+        callback(...args);
+      }
+    },
+  };
+
+  return { context, callbacks };
+}
+
 export function useStreamCallback<
   Type extends "onStart" | "onSuccess" | "onError"
 >(type: Type, callback: Exclude<StreamCallback[Type], undefined>) {

langserve/playground/src/useStreamLog.tsx

@@ -76,6 +76,7 @@ export function useStreamLog(callbacks: StreamCallback = {}) {
           setLatest(innerLatest);
         }
       },
+      openWhenHidden: true,
       onclose() {
         setController(null);
         successRef.current?.({ input, output: innerLatest?.final_output });

langserve/playground/src/utils/messages.ts

@@ -0,0 +1,7 @@
+export function getMessageContent(x: unknown) {
+  if (typeof x === "string") return x;
+  if (typeof x === "object" && x != null) {
+    if ("content" in x && typeof x.content === "string") return x.content;
+  }
+  return null;
+}

langserve/playground/src/utils/path.ts

@@ -2,11 +2,18 @@ function isAccessibleObject(x: unknown): x is Record<string | number, unknown> {
   return typeof x === "object" && x != null;
 }
 
+export function getNormalizedJsonPath(
+  path: string | number | Array<string | number>
+) {
+  return Array.isArray(path) ? path : [path];
+}
+
 export function traverseNaiveJsonPath(
   x: unknown,
   path: string | number | Array<string | number>
 ) {
-  const queue = Array.isArray(path) ? path : [path];
+  const queue = getNormalizedJsonPath(path);
+
   let tmp: unknown = x;
   while (queue.length > 0) {
     const first = queue.shift()!;

langserve/playground/src/utils/url.ts

@@ -1,5 +1,33 @@
-export function resolveApiUrl(path: string) {
-  let prefix = window.location.pathname.split("/playground")[0];
-  if (prefix.endsWith("/")) prefix = prefix.slice(0, -1);
-  return new URL(prefix + path, window.location.origin);
+import { decompressFromEncodedURIComponent } from "lz-string";
+
+export function getStateFromUrl(path: string) {
+  let configFromUrl = null;
+  let basePath = path;
+  if (basePath.endsWith("/")) {
+    basePath = basePath.slice(0, -1);
+  }
+
+  if (basePath.endsWith("/playground")) {
+    basePath = basePath.slice(0, -"/playground".length);
+  }
+
+  // check if we can omit the last segment
+  const [configHash, c, ...rest] = basePath.split("/").reverse();
+  if (c === "c") {
+    basePath = rest.reverse().join("/");
+    try {
+      configFromUrl = JSON.parse(decompressFromEncodedURIComponent(configHash));
+    } catch (error) {
+      console.error(error);
+    }
+  }
+  return { basePath, configFromUrl };
+}
+
+export function resolveApiUrl(path: string) {
+  const { basePath } = getStateFromUrl(window.location.href);
+  let prefix = new URL(basePath).pathname;
+  if (prefix.endsWith("/")) prefix = prefix.slice(0, -1);
+
+  return new URL(prefix + path, basePath);
 }

langserve/playground/vite.config.ts

@@ -8,7 +8,7 @@ export default defineConfig({
   plugins: [svgr(), react()],
   server: {
     proxy: {
-      "^/____LANGSERVE_BASE_URL.*/(config_schema|input_schema|stream_log)": {
+      "^/____LANGSERVE_BASE_URL.*/(config_schema|input_schema|stream_log|feedback)(/[a-zA-Z0-9-]*)?$": {
         target: "http://127.0.0.1:8000",
         changeOrigin: true,
         rewrite: (path) => path.replace("/____LANGSERVE_BASE_URL", ""),

langserve/playground/yarn.lock

@@ -1407,6 +1407,11 @@ chokidar@^3.5.3:
   optionalDependencies:
     fsevents "~2.3.2"
 
+client-only@^0.0.1:
+  version "0.0.1"
+  resolved "https://registry.yarnpkg.com/client-only/-/client-only-0.0.1.tgz#38bba5d403c41ab150bff64a95c85013cf73bca1"
+  integrity sha512-IV3Ou0jSMzZrd3pZ48nLkT9DA7Ag1pnPzaiQhpW7c3RbcqqzvzzVu+L8gfqMp/8IM2MQtSiqaCxrrcfu8I8rMA==
+
 clsx@^2.0.0:
   version "2.0.0"
   resolved "https://registry.yarnpkg.com/clsx/-/clsx-2.0.0.tgz#12658f3fd98fafe62075595a5c30e43d18f3d00b"
@@ -2620,6 +2625,14 @@ svg-parser@^2.0.4:
   resolved "https://registry.yarnpkg.com/svg-parser/-/svg-parser-2.0.4.tgz#fdc2e29e13951736140b76cb122c8ee6630eb6b5"
   integrity sha512-e4hG1hRwoOdRb37cIMSgzNsxyzKfayW6VOflrwvR+/bzrkyxY/31WkbgnQpgtrNp1SdpJvpUAGTa/ZoiPNDuRQ==
 
+swr@^2.2.4:
+  version "2.2.4"
+  resolved "https://registry.yarnpkg.com/swr/-/swr-2.2.4.tgz#03ec4c56019902fbdc904d78544bd7a9a6fa3f07"
+  integrity sha512-njiZ/4RiIhoOlAaLYDqwz5qH/KZXVilRLvomrx83HjzCWTfa+InyfAjv05PSFxnmLzZkNO9ZfvgoqzAaEI4sGQ==
+  dependencies:
+    client-only "^0.0.1"
+    use-sync-external-store "^1.2.0"
+
 tailwind-merge@^1.14.0:
   version "1.14.0"
   resolved "https://registry.yarnpkg.com/tailwind-merge/-/tailwind-merge-1.14.0.tgz#e677f55d864edc6794562c63f5001f45093cdb8b"
@@ -2751,6 +2764,11 @@ use-sidecar@^1.1.2:
     detect-node-es "^1.1.0"
     tslib "^2.0.0"
 
+use-sync-external-store@^1.2.0:
+  version "1.2.0"
+  resolved "https://registry.yarnpkg.com/use-sync-external-store/-/use-sync-external-store-1.2.0.tgz#7dbefd6ef3fe4e767a0cf5d7287aacfb5846928a"
+  integrity sha512-eEgnFxGQ1Ife9bzYs6VLi8/4X6CObHMw9Qr9tPY43iKwsPw8xE8+EFsf/2cFZ5S3esXgpWgtSCtLNS41F+sKPA==
+
 util-deprecate@^1.0.2:
   version "1.0.2"
   resolved "https://registry.yarnpkg.com/util-deprecate/-/util-deprecate-1.0.2.tgz#450d4dc9fa70de732762fbd2d4a28981419a0ccf"

langserve/pydantic.py

@@ -1,10 +0,0 @@
-import pydantic
-
-
-def _get_pydantic_version() -> int:
-    """Get the pydantic major version."""
-    return int(pydantic.__version__.split(".")[0])
-
-
-# Code is written to support both version 1 and 2
-PYDANTIC_MAJOR_VERSION = _get_pydantic_version()

langserve/pydantic_v1.py

@@ -0,0 +1,33 @@
+from importlib import metadata
+
+## Create namespaces for pydantic v1 and v2.
+# This code must stay at the top of the file before other modules may
+# attempt to import pydantic since it adds pydantic_v1 and pydantic_v2 to sys.modules.
+#
+# This hack is done for the following reasons:
+# * Langchain will attempt to remain compatible with both pydantic v1 and v2 since
+#   both dependencies and dependents may be stuck on either version of v1 or v2.
+# * Creating namespaces for pydantic v1 and v2 should allow us to write code that
+#   unambiguously uses either v1 or v2 API.
+# * This change is easier to roll out and roll back.
+
+try:
+    # F401: imported but unused
+    from pydantic.v1 import (  # noqa: F401
+        BaseModel,
+        Field,
+        ValidationError,
+        create_model,
+    )
+except ImportError:
+    from pydantic import BaseModel, Field, ValidationError, create_model  # noqa: F401
+
+
+# This is not a pydantic v1 thing, but it feels too small to create a new module for.
+
+PYDANTIC_VERSION = metadata.version("pydantic")
+
+try:
+    _PYDANTIC_MAJOR_VERSION: int = int(PYDANTIC_VERSION.split(".")[0])
+except metadata.PackageNotFoundError:
+    _PYDANTIC_MAJOR_VERSION = -1

langserve/schema.py

@@ -2,13 +2,9 @@ from datetime import datetime
 from typing import Dict, List, Optional, Union
 from uuid import UUID
 
-from langserve.pydantic import PYDANTIC_MAJOR_VERSION
+from pydantic import BaseModel  # Floats between v1 and v2
 
-if PYDANTIC_MAJOR_VERSION == 2:
-    from pydantic.v1 import BaseModel as BaseModelV1
-else:
-    from pydantic import BaseModel as BaseModelV1
-from pydantic import BaseModel
+from langserve.pydantic_v1 import BaseModel as BaseModelV1
 
 
 class CustomUserType(BaseModelV1):
@@ -100,6 +96,9 @@ class Feedback(BaseFeedback):
     Represents feedback given on an individual run
     """
 
+    id: UUID
+    """The unique ID of the feedback that was created."""
+
     created_at: datetime
     """The time the feedback was created."""
 

langserve/serialization.py

@@ -11,11 +11,11 @@ any information about the exception. This is done to prevent leaking
 sensitive information from the server to the client.
 """
 import abc
-import json
 import logging
 from functools import lru_cache
 from typing import Any, Dict, List, Union
 
+import orjson
 from langchain.prompts.base import StringPromptValue
 from langchain.prompts.chat import ChatPromptValueConcrete
 from langchain.schema.agent import AgentAction, AgentActionMessageLog, AgentFinish
@@ -39,14 +39,9 @@ from langchain.schema.output import (
     LLMResult,
 )
 
+from langserve.pydantic_v1 import BaseModel, ValidationError
 from langserve.validation import CallbackEvent
 
-try:
-    from pydantic.v1 import BaseModel, ValidationError
-except ImportError:
-    from pydantic import BaseModel, ValidationError
-
-
 logger = logging.getLogger(__name__)
 
 
@@ -88,14 +83,11 @@ class WellKnownLCObject(BaseModel):
     ]
 
 
-# Custom JSON Encoder
-class _LangChainEncoder(json.JSONEncoder):
-    """Custom JSON Encoder that can encode pydantic objects as well."""
-
-    def default(self, obj) -> Any:
-        if isinstance(obj, BaseModel):
-            return obj.dict()
-        return super().default(obj)
+def default(obj) -> Any:
+    """Default serialization for well known objects."""
+    if isinstance(obj, BaseModel):
+        return obj.dict()
+    return super().default(obj)
 
 
 def _decode_lc_objects(value: Any) -> Any:
@@ -154,11 +146,11 @@ class Serializer(abc.ABC):
         """Convert the given object to a JSON serializable object."""
 
     @abc.abstractmethod
-    def dumps(self, obj: Any) -> str:
+    def dumps(self, obj: Any) -> bytes:
         """Dump the given object as a JSON string."""
 
     @abc.abstractmethod
-    def loads(self, s: str) -> Any:
+    def loads(self, s: bytes) -> Any:
         """Load the given JSON string."""
 
     @abc.abstractmethod
@@ -169,18 +161,19 @@ class Serializer(abc.ABC):
 class WellKnownLCSerializer(Serializer):
     def dumpd(self, obj: Any) -> Any:
         """Convert the given object to a JSON serializable object."""
-        return json.loads(json.dumps(obj, cls=_LangChainEncoder))  # :*(
+        return orjson.loads(orjson.dumps(obj, default=default))
 
-    def dumps(self, obj: Any) -> str:
+    def dumps(self, obj: Any) -> bytes:
         """Dump the given object as a JSON string."""
-        return json.dumps(obj, cls=_LangChainEncoder)
+        return orjson.dumps(obj, default=default)
 
     def loadd(self, obj: Any) -> Any:
+        """Load the given object."""
         return _decode_lc_objects(obj)
 
-    def loads(self, s: str) -> Any:
+    def loads(self, s: bytes) -> Any:
         """Load the given JSON string."""
-        return self.loadd(json.loads(s))
+        return self.loadd(orjson.loads(s))
 
 
 def _project_top_level(model: BaseModel) -> Dict[str, Any]:

langserve/validation.py

@@ -26,6 +26,7 @@ from langchain.schema import (
     Generation,
     RunInfo,
 )
+from typing_extensions import Type
 
 from langserve.schema import BatchResponseMetadata, SingletonResponseMetadata
 
@@ -34,7 +35,6 @@ try:
 except ImportError:
     from pydantic import BaseModel, Field, create_model
 
-from typing_extensions import Type, TypedDict
 
 # Type that is either a python annotation or a pydantic model that can be
 # used to validate the input or output of a runnable.
@@ -46,7 +46,7 @@ Validator = Union[Type[BaseModel], type]
 def create_invoke_request_model(
     namespace: str,
     input_type: Validator,
-    config: TypedDict,
+    config: Type[BaseModel],
 ) -> Type[BaseModel]:
     """Create a pydantic model for the invoke request."""
     invoke_request_type = create_model(
@@ -77,7 +77,7 @@ def create_invoke_request_model(
 def create_stream_request_model(
     namespace: str,
     input_type: Validator,
-    config: TypedDict,
+    config: Type[BaseModel],
 ) -> Type[BaseModel]:
     """Create a pydantic model for the stream request."""
     stream_request_model = create_model(
@@ -108,7 +108,7 @@ def create_stream_request_model(
 def create_batch_request_model(
     namespace: str,
     input_type: Validator,
-    config: TypedDict,
+    config: Type[BaseModel],
 ) -> Type[BaseModel]:
     """Create a pydantic model for the batch request."""
     batch_request_type = create_model(
@@ -140,7 +140,7 @@ def create_batch_request_model(
 def create_stream_log_request_model(
     namespace: str,
     input_type: Validator,
-    config: TypedDict,
+    config: Type[BaseModel],
 ) -> Type[BaseModel]:
     """Create a pydantic model for the invoke request."""
     stream_log_request = create_model(
@@ -195,6 +195,64 @@ def create_stream_log_request_model(
     return stream_log_request
 
 
+def create_stream_events_request_model(
+    namespace: str,
+    input_type: Validator,
+    config: Type[BaseModel],
+) -> Type[BaseModel]:
+    """Create a pydantic model for the stream events request."""
+    stream_events_request = create_model(
+        f"{namespace}StreamEventsRequest",
+        input=(input_type, ...),
+        config=(config, Field(default_factory=dict)),
+        include_names=(
+            Optional[Sequence[str]],
+            Field(
+                None,
+                description="If specified, filter to runnables with matching names",
+            ),
+        ),
+        include_types=(
+            Optional[Sequence[str]],
+            Field(
+                None,
+                description="If specified, filter to runnables with matching types",
+            ),
+        ),
+        include_tags=(
+            Optional[Sequence[str]],
+            Field(
+                None,
+                description="If specified, filter to runnables with matching tags",
+            ),
+        ),
+        exclude_names=(
+            Optional[Sequence[str]],
+            Field(
+                None,
+                description="If specified, exclude runnables with matching names",
+            ),
+        ),
+        exclude_types=(
+            Optional[Sequence[str]],
+            Field(
+                None,
+                description="If specified, exclude runnables with matching types",
+            ),
+        ),
+        exclude_tags=(
+            Optional[Sequence[str]],
+            Field(
+                None,
+                description="If specified, exclude runnables with matching tags",
+            ),
+        ),
+        kwargs=(dict, Field(default_factory=dict)),
+    )
+    stream_events_request.update_forward_refs()
+    return stream_events_request
+
+
 class InvokeBaseResponse(BaseModel):
     """Base class for invoke request."""
 
@@ -308,6 +366,17 @@ class StreamLogParameters(BaseModel):
     exclude_tags: Optional[Sequence[str]] = None
 
 
+class StreamEventsParameters(BaseModel):
+    """Shallow validator for Stream Events Request."""
+
+    include_names: Optional[Sequence[str]] = None
+    include_types: Optional[Sequence[str]] = None
+    include_tags: Optional[Sequence[str]] = None
+    exclude_names: Optional[Sequence[str]] = None
+    exclude_types: Optional[Sequence[str]] = None
+    exclude_tags: Optional[Sequence[str]] = None
+
+
 # Pydantic validators for callback events
 # These objects may have a slightly different shape than the callback events
 # used internally in langchain because they represent a serialized version

pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "langserve"
-version = "0.0.24"
+version = "0.0.44"
 description = ""
 readme = "README.md"
 authors = ["LangChain"]
@@ -12,11 +12,12 @@ include = ["langserve/playground/dist/**/*"]
 [tool.poetry.dependencies]
 python = "^3.8.1"
 httpx = ">=0.23.0" # May be able to decrease this version
-fastapi = {version = ">=0.90.1", optional = true}
+fastapi = {version = ">=0.90.1,<1", optional = true}
 sse-starlette = {version = "^1.3.0", optional = true}
 httpx-sse = {version = ">=0.3.1", optional = true}
 pydantic = ">=1"
-langchain = ">=0.0.322"
+langchain = ">=0.0.333"
+orjson = ">=2"
 
 [tool.poetry.group.dev.dependencies]
 jupyterlab = "^3.6.1"
@@ -37,6 +38,7 @@ pytest-asyncio = "^0.21.1"
 pytest-mock = "^3.11.1"
 pytest-socket = "^0.6.0"
 pytest-watch = "^4.2.0"
+pytest-timeout = "^2.2.0"
 
 [tool.poetry.group.examples.dependencies]
 openai = "^0.28.0"
@@ -83,3 +85,15 @@ omit = [
 requires = ["poetry-core"]
 build-backend = "poetry.core.masonry.api"
 
+[tool.pytest.ini_options]
+# --strict-markers will raise errors on unknown marks.
+# https://docs.pytest.org/en/7.1.x/how-to/mark.html#raising-errors-on-unknown-marks
+#
+# https://docs.pytest.org/en/7.1.x/reference/reference.html
+# --strict-config       any warnings encountered while parsing the `pytest`
+#                       section of the configuration file raise errors.
+addopts = "--strict-markers --strict-config --durations=5 -vv"
+# Global timeout for all tests. There shuold be a good reason for a test to
+# take more than 5 seconds
+timeout = 5
+asyncio_mode = "auto"

tests/unit_tests/test_callbacks.py

@@ -1,11 +1,8 @@
 import uuid
 
-import pytest
-
 from langserve.callbacks import AsyncEventAggregatorCallback, replace_uuids
 
 
-@pytest.mark.asyncio
 async def test_event_aggregator() -> None:
     """Test that the event aggregator is aggregating events."""
 

tests/unit_tests/test_playground.py

@@ -0,0 +1,24 @@
+import pytest
+
+from langserve.playground import _get_mimetype
+
+
+@pytest.mark.parametrize(
+    "file_extension, expected_mimetype",
+    [
+        ("js", "application/javascript"),
+        ("css", "text/css"),
+        ("htm", "text/html"),
+        ("html", "text/html"),
+        ("txt", "text/plain"),  # An example of an unknown extension using guess_type
+    ],
+)
+def test_get_mimetype(file_extension: str, expected_mimetype: str) -> None:
+    # Create a filename with the given extension
+    filename = f"test_file.{file_extension}"
+
+    # Call the _get_mimetype function with the test filename
+    mimetype = _get_mimetype(filename)
+
+    # Check if the returned mimetype matches the expected one
+    assert mimetype == expected_mimetype

tests/unit_tests/test_serialization.py

@@ -1,4 +1,6 @@
+import datetime
 import uuid
+from enum import Enum
 from typing import Any
 
 import pytest
@@ -49,7 +51,7 @@ def test_serialization(data: Any) -> None:
     # Test encoding
     lc_serializer = WellKnownLCSerializer()
 
-    assert isinstance(lc_serializer.dumps(data), str)
+    assert isinstance(lc_serializer.dumps(data), bytes)
     # Translate to python primitives and load back into object
     assert lc_serializer.loadd(lc_serializer.dumpd(data)) == data
     # Test simple equality (does not include pydantic class names)
@@ -119,3 +121,65 @@ def _get_full_representation(data: Any) -> Any:
 def test_decode_events(data: Any, expected: Any) -> None:
     """Test decoding events."""
     assert load_events(data) == expected
+
+
+class SimpleEnum(Enum):
+    a = "a"
+    b = "b"
+
+
+class SimpleModel(BaseModel):
+    x: int
+    y: SimpleEnum
+    z: uuid.UUID
+    dt: datetime.datetime
+    d: datetime.date
+    t: datetime.time
+
+
+@pytest.mark.parametrize(
+    "obj, expected",
+    [
+        ({"key": "value"}, {"key": "value"}),
+        ([1, 2, 3], [1, 2, 3]),
+        (123, 123),
+        (uuid.UUID(int=1), "00000000-0000-0000-0000-000000000001"),
+        (datetime.datetime(2020, 1, 1), "2020-01-01T00:00:00"),
+        (datetime.date(2020, 1, 1), "2020-01-01"),
+        (datetime.time(0, 0, 0), "00:00:00"),
+        (datetime.time(0, 0, 0, 1), "00:00:00.000001"),
+        (SimpleEnum.a, "a"),
+        (
+            SimpleModel(
+                x=1,
+                y=SimpleEnum.a,
+                z=uuid.UUID(int=1),
+                dt=datetime.datetime(2020, 1, 1),
+                d=datetime.date(2020, 1, 1),
+                t=datetime.time(0, 0, 0),
+            ),
+            {
+                "x": 1,
+                "y": "a",
+                "z": "00000000-0000-0000-0000-000000000001",
+                "dt": "2020-01-01T00:00:00",
+                "d": "2020-01-01",
+                "t": "00:00:00",
+            },
+        ),
+        ("string", "string"),
+        (True, True),
+        (None, None),
+    ],
+)
+def test_encoding_of_well_known_types(obj: Any, expected: str) -> None:
+    """Test encoding of well known types.
+
+    This tests verifies that our custom serializer is able to encode some well
+    known types; e.g., uuid, datetime, date, time
+
+    It doesn't handle types like Decimal or frozenset etc just yet while we determine
+    how to roll out a more complete solution.
+    """
+    lc_serializer = WellKnownLCSerializer()
+    assert lc_serializer.dumpd(obj) == expected

tests/unit_tests/test_validation.py

@@ -6,7 +6,7 @@ from fastapi import Request
 from langchain.prompts import PromptTemplate
 from langchain.schema.runnable.utils import ConfigurableField
 
-from langserve.server import _unpack_request_config
+from langserve.api_handler import _unpack_request_config
 
 try:
     from pydantic.v1 import BaseModel, ValidationError
@@ -140,7 +140,7 @@ def test_validation(test_case) -> None:
             model(**test_case)
 
 
-def test_invoke_request_with_runnables() -> None:
+async def test_invoke_request_with_runnables() -> None:
     """Test that the invoke request model is created correctly."""
     runnable = PromptTemplate.from_template("say hello to {name}").configurable_fields(
         template=ConfigurableField(
@@ -153,14 +153,15 @@ def test_invoke_request_with_runnables() -> None:
     Model = create_invoke_request_model("", runnable.input_schema, config)
 
     assert (
-        _unpack_request_config(
+        await _unpack_request_config(
             Model(
                 input={"name": "bob"},
             ).config,
-            keys=[],
+            config_keys=[],
             model=config,
             request=MagicMock(Request),
             per_req_config_modifier=lambda x, y: x,
+            server_config=None,
         )
         == {}
     )
@@ -182,12 +183,13 @@ def test_invoke_request_with_runnables() -> None:
         "template": "goodbye {name}",
     }
 
-    assert _unpack_request_config(
+    assert await _unpack_request_config(
         request.config,
-        keys=["configurable"],
+        config_keys=["configurable"],
         model=config,
         request=MagicMock(Request),
         per_req_config_modifier=lambda x, y: x,
+        server_config=None,
     ) == {
         "configurable": {"template": "goodbye {name}"},
     }

tests/unit_tests/utils/llms.py

@@ -0,0 +1,244 @@
+"""Fake Chat Model wrapper for testing purposes."""
+import asyncio
+import re
+import time
+from typing import (
+    Any,
+    AsyncIterator,
+    Iterator,
+    List,
+    Mapping,
+    Optional,
+    cast,
+)
+
+from langchain_core.callbacks.manager import (
+    AsyncCallbackManagerForLLMRun,
+    CallbackManagerForLLMRun,
+)
+from langchain_core.language_models import LLM
+from langchain_core.language_models.chat_models import BaseChatModel
+from langchain_core.messages import (
+    AIMessage,
+    AIMessageChunk,
+    BaseMessage,
+)
+from langchain_core.outputs import ChatGeneration, ChatGenerationChunk, ChatResult
+from langchain_core.runnables import run_in_executor
+
+
+class GenericFakeChatModel(BaseChatModel):
+    """A generic fake chat model that can be used to test the chat model interface.
+
+    * Chat model should be usable in both sync and async tests
+    * Invokes on_llm_new_token to allow for testing of callback related code for new
+      tokens.
+    * Includes logic to break messages into message chunk to facilitate testing of
+      streaming.
+    """
+
+    messages: Iterator[AIMessage]
+    """Get an iterator over messages.
+
+    This can be expanded to accept other types like Callables / dicts / strings
+    to make the interface more generic if needed.
+
+    Note: if you want to pass a list, you can use `iter` to convert it to an iterator.
+
+    Please note that streaming is not implemented yet. We should try to implement it
+    in the future by delegating to invoke and then breaking the resulting output
+    into message chunks.
+    """
+    delay_per_token: Optional[float] = None
+    """Delay per token in seconds.
+    
+    If specified, we will sleep for this amount of time after each token when
+    streaming.
+    """
+
+    def _generate(
+        self,
+        messages: List[BaseMessage],
+        stop: Optional[List[str]] = None,
+        run_manager: Optional[CallbackManagerForLLMRun] = None,
+        **kwargs: Any,
+    ) -> ChatResult:
+        """Top Level call"""
+        message = next(self.messages)
+        generation = ChatGeneration(message=message)
+        return ChatResult(generations=[generation])
+
+    def _stream(
+        self,
+        messages: List[BaseMessage],
+        stop: Optional[List[str]] = None,
+        run_manager: Optional[CallbackManagerForLLMRun] = None,
+        **kwargs: Any,
+    ) -> Iterator[ChatGenerationChunk]:
+        for chunk in self._stream_implementation(
+            messages, stop=stop, run_manager=run_manager, **kwargs
+        ):
+            if self.delay_per_token:
+                time.sleep(self.delay_per_token)
+            yield chunk
+
+    def _stream_implementation(
+        self,
+        messages: List[BaseMessage],
+        stop: Optional[List[str]] = None,
+        run_manager: Optional[CallbackManagerForLLMRun] = None,
+        **kwargs: Any,
+    ) -> Iterator[ChatGenerationChunk]:
+        """Stream the output of the model."""
+        chat_result = self._generate(
+            messages, stop=stop, run_manager=run_manager, **kwargs
+        )
+        if not isinstance(chat_result, ChatResult):
+            raise ValueError(
+                f"Expected generate to return a ChatResult, "
+                f"but got {type(chat_result)} instead."
+            )
+
+        message = chat_result.generations[0].message
+
+        if not isinstance(message, AIMessage):
+            raise ValueError(
+                f"Expected invoke to return an AIMessage, "
+                f"but got {type(message)} instead."
+            )
+
+        content = message.content
+
+        if content:
+            # Use a regular expression to split on whitespace with a capture group
+            # so that we can preserve the whitespace in the output.
+            assert isinstance(content, str)
+            content_chunks = cast(List[str], re.split(r"(\s)", content))
+
+            for token in content_chunks:
+                chunk = ChatGenerationChunk(message=AIMessageChunk(content=token))
+                yield chunk
+                if run_manager:
+                    run_manager.on_llm_new_token(token, chunk=chunk)
+
+        if message.additional_kwargs:
+            for key, value in message.additional_kwargs.items():
+                # We should further break down the additional kwargs into chunks
+                # Special case for function call
+                if key == "function_call":
+                    for fkey, fvalue in value.items():
+                        if isinstance(fvalue, str):
+                            # Break function call by `,`
+                            fvalue_chunks = cast(List[str], re.split(r"(,)", fvalue))
+                            for fvalue_chunk in fvalue_chunks:
+                                chunk = ChatGenerationChunk(
+                                    message=AIMessageChunk(
+                                        content="",
+                                        additional_kwargs={
+                                            "function_call": {fkey: fvalue_chunk}
+                                        },
+                                    )
+                                )
+                                yield chunk
+                                if run_manager:
+                                    run_manager.on_llm_new_token(
+                                        "",
+                                        chunk=chunk,  # No token for function call
+                                    )
+                        else:
+                            chunk = ChatGenerationChunk(
+                                message=AIMessageChunk(
+                                    content="",
+                                    additional_kwargs={"function_call": {fkey: fvalue}},
+                                )
+                            )
+                            yield chunk
+                            if run_manager:
+                                run_manager.on_llm_new_token(
+                                    "",
+                                    chunk=chunk,  # No token for function call
+                                )
+                else:
+                    chunk = ChatGenerationChunk(
+                        message=AIMessageChunk(
+                            content="", additional_kwargs={key: value}
+                        )
+                    )
+                    yield chunk
+                    if run_manager:
+                        run_manager.on_llm_new_token(
+                            "",
+                            chunk=chunk,  # No token for function call
+                        )
+
+    async def _astream(
+        self,
+        messages: List[BaseMessage],
+        stop: Optional[List[str]] = None,
+        run_manager: Optional[AsyncCallbackManagerForLLMRun] = None,
+        **kwargs: Any,
+    ) -> AsyncIterator[ChatGenerationChunk]:
+        """Stream the output of the model."""
+        result = await run_in_executor(
+            None,
+            self._stream,
+            messages,
+            stop=stop,
+            run_manager=run_manager.get_sync() if run_manager else None,
+            **kwargs,
+        )
+        for chunk in result:
+            if self.delay_per_token:
+                await asyncio.sleep(self.delay_per_token)
+            yield chunk
+
+    @property
+    def _llm_type(self) -> str:
+        return "generic-fake-chat-model"
+
+
+class FakeListLLM(LLM):
+    """Fake LLM for testing purposes."""
+
+    responses: List[str]
+    sleep: Optional[float] = None
+    i: int = 0
+
+    @property
+    def _llm_type(self) -> str:
+        """Return type of llm."""
+        return "fake-list"
+
+    def _call(
+        self,
+        prompt: str,
+        stop: Optional[List[str]] = None,
+        run_manager: Optional[CallbackManagerForLLMRun] = None,
+        **kwargs: Any,
+    ) -> str:
+        """Return next response"""
+        response = self.responses[self.i]
+        if self.i < len(self.responses) - 1:
+            self.i += 1
+        else:
+            self.i = 0
+        return response
+
+    async def _acall(
+        self,
+        prompt: str,
+        stop: Optional[List[str]] = None,
+        run_manager: Optional[AsyncCallbackManagerForLLMRun] = None,
+        **kwargs: Any,
+    ) -> str:
+        """Return next response"""
+        response = self.responses[self.i]
+        if self.i < len(self.responses) - 1:
+            self.i += 1
+        else:
+            self.i = 0
+        return response
+
+    @property
+    def _identifying_params(self) -> Mapping[str, Any]:
+        return {"responses": self.responses}

tests/unit_tests/utils/test_fake_chat_model.py

@@ -0,0 +1,185 @@
+"""Tests for verifying that testing utility code works as expected."""
+from itertools import cycle
+from typing import Any, Dict, List, Optional, Union
+from uuid import UUID
+
+from langchain_core.callbacks.base import AsyncCallbackHandler
+from langchain_core.messages import AIMessage, AIMessageChunk, BaseMessage
+from langchain_core.outputs import ChatGenerationChunk, GenerationChunk
+
+from tests.unit_tests.utils.llms import GenericFakeChatModel
+
+
+def test_generic_fake_chat_model_invoke() -> None:
+    # Will alternate between responding with hello and goodbye
+    infinite_cycle = cycle([AIMessage(content="hello"), AIMessage(content="goodbye")])
+    model = GenericFakeChatModel(messages=infinite_cycle)
+    response = model.invoke("meow")
+    assert response == AIMessage(content="hello")
+    response = model.invoke("kitty")
+    assert response == AIMessage(content="goodbye")
+    response = model.invoke("meow")
+    assert response == AIMessage(content="hello")
+
+
+async def test_generic_fake_chat_model_ainvoke() -> None:
+    # Will alternate between responding with hello and goodbye
+    infinite_cycle = cycle([AIMessage(content="hello"), AIMessage(content="goodbye")])
+    model = GenericFakeChatModel(messages=infinite_cycle)
+    response = await model.ainvoke("meow")
+    assert response == AIMessage(content="hello")
+    response = await model.ainvoke("kitty")
+    assert response == AIMessage(content="goodbye")
+    response = await model.ainvoke("meow")
+    assert response == AIMessage(content="hello")
+
+
+async def test_generic_fake_chat_model_stream() -> None:
+    """Test streaming."""
+    infinite_cycle = cycle(
+        [
+            AIMessage(content="hello goodbye"),
+        ]
+    )
+    model = GenericFakeChatModel(messages=infinite_cycle)
+    chunks = [chunk async for chunk in model.astream("meow")]
+    assert chunks == [
+        AIMessageChunk(content="hello"),
+        AIMessageChunk(content=" "),
+        AIMessageChunk(content="goodbye"),
+    ]
+
+    chunks = [chunk for chunk in model.stream("meow")]
+    assert chunks == [
+        AIMessageChunk(content="hello"),
+        AIMessageChunk(content=" "),
+        AIMessageChunk(content="goodbye"),
+    ]
+
+    # Test streaming of additional kwargs.
+    # Relying on insertion order of the additional kwargs dict
+    message = AIMessage(content="", additional_kwargs={"foo": 42, "bar": 24})
+    model = GenericFakeChatModel(messages=cycle([message]))
+    chunks = [chunk async for chunk in model.astream("meow")]
+    assert chunks == [
+        AIMessageChunk(content="", additional_kwargs={"foo": 42}),
+        AIMessageChunk(content="", additional_kwargs={"bar": 24}),
+    ]
+
+    message = AIMessage(
+        content="",
+        additional_kwargs={
+            "function_call": {
+                "name": "move_file",
+                "arguments": '{\n  "source_path": "foo",\n  "'
+                'destination_path": "bar"\n}',
+            }
+        },
+    )
+    model = GenericFakeChatModel(messages=cycle([message]))
+    chunks = [chunk async for chunk in model.astream("meow")]
+
+    assert chunks == [
+        AIMessageChunk(
+            content="", additional_kwargs={"function_call": {"name": "move_file"}}
+        ),
+        AIMessageChunk(
+            content="",
+            additional_kwargs={
+                "function_call": {"arguments": '{\n  "source_path": "foo"'}
+            },
+        ),
+        AIMessageChunk(
+            content="", additional_kwargs={"function_call": {"arguments": ","}}
+        ),
+        AIMessageChunk(
+            content="",
+            additional_kwargs={
+                "function_call": {"arguments": '\n  "destination_path": "bar"\n}'}
+            },
+        ),
+    ]
+
+    accumulate_chunks = None
+    for chunk in chunks:
+        if accumulate_chunks is None:
+            accumulate_chunks = chunk
+        else:
+            accumulate_chunks += chunk
+
+    assert accumulate_chunks == AIMessageChunk(
+        content="",
+        additional_kwargs={
+            "function_call": {
+                "name": "move_file",
+                "arguments": '{\n  "source_path": "foo",\n  "'
+                'destination_path": "bar"\n}',
+            }
+        },
+    )
+
+
+async def test_generic_fake_chat_model_astream_log() -> None:
+    """Test streaming."""
+    infinite_cycle = cycle([AIMessage(content="hello goodbye")])
+    model = GenericFakeChatModel(messages=infinite_cycle)
+    log_patches = [
+        log_patch async for log_patch in model.astream_log("meow", diff=False)
+    ]
+    final = log_patches[-1]
+    assert final.state["streamed_output"] == [
+        AIMessageChunk(content="hello"),
+        AIMessageChunk(content=" "),
+        AIMessageChunk(content="goodbye"),
+    ]
+
+
+async def test_callback_handlers() -> None:
+    """Verify that model is implemented correctly with handlers working."""
+
+    class MyCustomAsyncHandler(AsyncCallbackHandler):
+        def __init__(self, store: List[str]) -> None:
+            self.store = store
+
+        async def on_chat_model_start(
+            self,
+            serialized: Dict[str, Any],
+            messages: List[List[BaseMessage]],
+            *,
+            run_id: UUID,
+            parent_run_id: Optional[UUID] = None,
+            tags: Optional[List[str]] = None,
+            metadata: Optional[Dict[str, Any]] = None,
+            **kwargs: Any,
+        ) -> Any:
+            # Do nothing
+            # Required to implement since this is an abstract method
+            pass
+
+        async def on_llm_new_token(
+            self,
+            token: str,
+            *,
+            chunk: Optional[Union[GenerationChunk, ChatGenerationChunk]] = None,
+            run_id: UUID,
+            parent_run_id: Optional[UUID] = None,
+            tags: Optional[List[str]] = None,
+            **kwargs: Any,
+        ) -> None:
+            self.store.append(token)
+
+    infinite_cycle = cycle(
+        [
+            AIMessage(content="hello goodbye"),
+        ]
+    )
+    model = GenericFakeChatModel(messages=infinite_cycle)
+    tokens: List[str] = []
+    # New model
+    results = list(model.stream("meow", {"callbacks": [MyCustomAsyncHandler(tokens)]}))
+    assert results == [
+        AIMessageChunk(content="hello"),
+        AIMessageChunk(content=" "),
+        AIMessageChunk(content="goodbye"),
+    ]
+    assert tokens == ["hello", " ", "goodbye"]

tests/unit_tests/utils/tracer.py

@@ -1,62 +1,10 @@
-from typing import Any, Dict, List, Mapping, Optional
+from typing import Dict, List
 from uuid import UUID
 
-from langchain.callbacks.manager import (
-    AsyncCallbackManagerForLLMRun,
-    CallbackManagerForLLMRun,
-)
 from langchain.callbacks.tracers.base import BaseTracer
-from langchain.llms.base import LLM
 from langsmith.schemas import Run
 
 
-class FakeListLLM(LLM):
-    """Fake LLM for testing purposes."""
-
-    responses: List[str]
-    sleep: Optional[float] = None
-    i: int = 0
-
-    @property
-    def _llm_type(self) -> str:
-        """Return type of llm."""
-        return "fake-list"
-
-    def _call(
-        self,
-        prompt: str,
-        stop: Optional[List[str]] = None,
-        run_manager: Optional[CallbackManagerForLLMRun] = None,
-        **kwargs: Any,
-    ) -> str:
-        """Return next response"""
-        response = self.responses[self.i]
-        if self.i < len(self.responses) - 1:
-            self.i += 1
-        else:
-            self.i = 0
-        return response
-
-    async def _acall(
-        self,
-        prompt: str,
-        stop: Optional[List[str]] = None,
-        run_manager: Optional[AsyncCallbackManagerForLLMRun] = None,
-        **kwargs: Any,
-    ) -> str:
-        """Return next response"""
-        response = self.responses[self.i]
-        if self.i < len(self.responses) - 1:
-            self.i += 1
-        else:
-            self.i = 0
-        return response
-
-    @property
-    def _identifying_params(self) -> Mapping[str, Any]:
-        return {"responses": self.responses}
-
-
 class FakeTracer(BaseTracer):
     """Fake tracer that records LangChain execution.