{
 "cells": [
  {
   "cell_type": "raw",
   "id": "366a0e68-fd67-4fe5-a292-5c33733339ea",
   "metadata": {},
   "source": [
    "---\n",
    "sidebar_position: 0\n",
    "title: Interface\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a9acd2e",
   "metadata": {},
   "source": [
    "To make it as easy as possible to create custom chains, we've implemented a [\"Runnable\"](https://api.python.langchain.com/en/latest/schema/langchain.schema.runnable.base.Runnable.html#langchain.schema.runnable.base.Runnable) protocol. The `Runnable` protocol is implemented for most components. \n",
    "This is a standard interface, which makes it easy to define custom chains as well as invoke them in a standard way. \n",
    "The standard interface includes:\n",
    "\n",
    "- [`stream`](#stream): stream back chunks of the response\n",
    "- [`invoke`](#invoke): call the chain on an input\n",
    "- [`batch`](#batch): call the chain on a list of inputs\n",
    "\n",
    "These also have corresponding async methods:\n",
    "\n",
    "- [`astream`](#async-stream): stream back chunks of the response async\n",
    "- [`ainvoke`](#async-invoke): call the chain on an input async\n",
    "- [`abatch`](#async-batch): call the chain on a list of inputs async\n",
    "- [`astream_log`](#async-stream-intermediate-steps): stream back intermediate steps as they happen, in addition to the final response\n",
    "\n",
    "The **input type** and **output type** varies by component:\n",
    "\n",
    "| Component | Input Type | Output Type |\n",
    "| --- | --- | --- |\n",
    "| Prompt | Dictionary | PromptValue |\n",
    "| ChatModel | Single string, list of chat messages or a PromptValue | ChatMessage |\n",
    "| LLM | Single string, list of chat messages or a PromptValue | String |\n",
    "| OutputParser | The output of an LLM or ChatModel | Depends on the parser |\n",
    "| Retriever | Single string | List of Documents |\n",
    "| Tool | Single string or dictionary, depending on the tool | Depends on the tool |\n",
    "\n",
    "\n",
    "All runnables expose input and output **schemas** to inspect the inputs and outputs:\n",
    "- [`input_schema`](#input-schema): an input Pydantic model auto-generated from the structure of the Runnable\n",
    "- [`output_schema`](#output-schema): an output Pydantic model auto-generated from the structure of the Runnable\n",
    "\n",
    "Let's take a look at these methods. To do so, we'll create a super simple PromptTemplate + ChatModel chain."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "466b65b3",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.chat_models import ChatOpenAI\n",
    "from langchain.prompts import ChatPromptTemplate\n",
    "\n",
    "model = ChatOpenAI()\n",
    "prompt = ChatPromptTemplate.from_template(\"tell me a joke about {topic}\")\n",
    "chain = prompt | model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5cccdf0b-2d89-4f74-9530-bf499610e9a5",
   "metadata": {},
   "source": [
    "## Input Schema\n",
    "\n",
    "A description of the inputs accepted by a Runnable.\n",
    "This is a Pydantic model dynamically generated from the structure of any Runnable.\n",
    "You can call `.schema()` on it to obtain a JSONSchema representation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "25e146d4-60da-40a2-9026-b5dfee106a3f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'title': 'PromptInput',\n",
       " 'type': 'object',\n",
       " 'properties': {'topic': {'title': 'Topic', 'type': 'string'}}}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The input schema of the chain is the input schema of its first part, the prompt.\n",
    "chain.input_schema.schema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "ad130546-4c14-4f6c-95af-c56ea19b12ac",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'title': 'PromptInput',\n",
       " 'type': 'object',\n",
       " 'properties': {'topic': {'title': 'Topic', 'type': 'string'}}}"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prompt.input_schema.schema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "49d34744-d6db-4fdf-a0d6-261522b7f251",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'title': 'ChatOpenAIInput',\n",
       " 'anyOf': [{'type': 'string'},\n",
       "  {'$ref': '#/definitions/StringPromptValue'},\n",
       "  {'$ref': '#/definitions/ChatPromptValueConcrete'},\n",
       "  {'type': 'array',\n",
       "   'items': {'anyOf': [{'$ref': '#/definitions/AIMessage'},\n",
       "     {'$ref': '#/definitions/HumanMessage'},\n",
       "     {'$ref': '#/definitions/ChatMessage'},\n",
       "     {'$ref': '#/definitions/SystemMessage'},\n",
       "     {'$ref': '#/definitions/FunctionMessage'}]}}],\n",
       " 'definitions': {'StringPromptValue': {'title': 'StringPromptValue',\n",
       "   'description': 'String prompt value.',\n",
       "   'type': 'object',\n",
       "   'properties': {'text': {'title': 'Text', 'type': 'string'},\n",
       "    'type': {'title': 'Type',\n",
       "     'default': 'StringPromptValue',\n",
       "     'enum': ['StringPromptValue'],\n",
       "     'type': 'string'}},\n",
       "   'required': ['text']},\n",
       "  'AIMessage': {'title': 'AIMessage',\n",
       "   'description': 'A Message from an AI.',\n",
       "   'type': 'object',\n",
       "   'properties': {'content': {'title': 'Content', 'type': 'string'},\n",
       "    'additional_kwargs': {'title': 'Additional Kwargs', 'type': 'object'},\n",
       "    'type': {'title': 'Type',\n",
       "     'default': 'ai',\n",
       "     'enum': ['ai'],\n",
       "     'type': 'string'},\n",
       "    'example': {'title': 'Example', 'default': False, 'type': 'boolean'}},\n",
       "   'required': ['content']},\n",
       "  'HumanMessage': {'title': 'HumanMessage',\n",
       "   'description': 'A Message from a human.',\n",
       "   'type': 'object',\n",
       "   'properties': {'content': {'title': 'Content', 'type': 'string'},\n",
       "    'additional_kwargs': {'title': 'Additional Kwargs', 'type': 'object'},\n",
       "    'type': {'title': 'Type',\n",
       "     'default': 'human',\n",
       "     'enum': ['human'],\n",
       "     'type': 'string'},\n",
       "    'example': {'title': 'Example', 'default': False, 'type': 'boolean'}},\n",
       "   'required': ['content']},\n",
       "  'ChatMessage': {'title': 'ChatMessage',\n",
       "   'description': 'A Message that can be assigned an arbitrary speaker (i.e. role).',\n",
       "   'type': 'object',\n",
       "   'properties': {'content': {'title': 'Content', 'type': 'string'},\n",
       "    'additional_kwargs': {'title': 'Additional Kwargs', 'type': 'object'},\n",
       "    'type': {'title': 'Type',\n",
       "     'default': 'chat',\n",
       "     'enum': ['chat'],\n",
       "     'type': 'string'},\n",
       "    'role': {'title': 'Role', 'type': 'string'}},\n",
       "   'required': ['content', 'role']},\n",
       "  'SystemMessage': {'title': 'SystemMessage',\n",
       "   'description': 'A Message for priming AI behavior, usually passed in as the first of a sequence\\nof input messages.',\n",
       "   'type': 'object',\n",
       "   'properties': {'content': {'title': 'Content', 'type': 'string'},\n",
       "    'additional_kwargs': {'title': 'Additional Kwargs', 'type': 'object'},\n",
       "    'type': {'title': 'Type',\n",
       "     'default': 'system',\n",
       "     'enum': ['system'],\n",
       "     'type': 'string'}},\n",
       "   'required': ['content']},\n",
       "  'FunctionMessage': {'title': 'FunctionMessage',\n",
       "   'description': 'A Message for passing the result of executing a function back to a model.',\n",
       "   'type': 'object',\n",
       "   'properties': {'content': {'title': 'Content', 'type': 'string'},\n",
       "    'additional_kwargs': {'title': 'Additional Kwargs', 'type': 'object'},\n",
       "    'type': {'title': 'Type',\n",
       "     'default': 'function',\n",
       "     'enum': ['function'],\n",
       "     'type': 'string'},\n",
       "    'name': {'title': 'Name', 'type': 'string'}},\n",
       "   'required': ['content', 'name']},\n",
       "  'ChatPromptValueConcrete': {'title': 'ChatPromptValueConcrete',\n",
       "   'description': 'Chat prompt value which explicitly lists out the message types it accepts.\\nFor use in external schemas.',\n",
       "   'type': 'object',\n",
       "   'properties': {'messages': {'title': 'Messages',\n",
       "     'type': 'array',\n",
       "     'items': {'anyOf': [{'$ref': '#/definitions/AIMessage'},\n",
       "       {'$ref': '#/definitions/HumanMessage'},\n",
       "       {'$ref': '#/definitions/ChatMessage'},\n",
       "       {'$ref': '#/definitions/SystemMessage'},\n",
       "       {'$ref': '#/definitions/FunctionMessage'}]}},\n",
       "    'type': {'title': 'Type',\n",
       "     'default': 'ChatPromptValueConcrete',\n",
       "     'enum': ['ChatPromptValueConcrete'],\n",
       "     'type': 'string'}},\n",
       "   'required': ['messages']}}}"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.input_schema.schema()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5059a5dc-d544-4add-85bd-78a3f2b78b9a",
   "metadata": {},
   "source": [
    "## Output Schema\n",
    "\n",
    "A description of the outputs produced by a Runnable.\n",
    "This is a Pydantic model dynamically generated from the structure of any Runnable.\n",
    "You can call `.schema()` on it to obtain a JSONSchema representation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "a0e41fd3-77d8-4911-af6a-d4d3aad5f77b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'title': 'ChatOpenAIOutput',\n",
       " 'anyOf': [{'$ref': '#/definitions/HumanMessage'},\n",
       "  {'$ref': '#/definitions/AIMessage'},\n",
       "  {'$ref': '#/definitions/ChatMessage'},\n",
       "  {'$ref': '#/definitions/FunctionMessage'},\n",
       "  {'$ref': '#/definitions/SystemMessage'}],\n",
       " 'definitions': {'HumanMessage': {'title': 'HumanMessage',\n",
       "   'description': 'A Message from a human.',\n",
       "   'type': 'object',\n",
       "   'properties': {'content': {'title': 'Content', 'type': 'string'},\n",
       "    'additional_kwargs': {'title': 'Additional Kwargs', 'type': 'object'},\n",
       "    'type': {'title': 'Type',\n",
       "     'default': 'human',\n",
       "     'enum': ['human'],\n",
       "     'type': 'string'},\n",
       "    'example': {'title': 'Example', 'default': False, 'type': 'boolean'}},\n",
       "   'required': ['content']},\n",
       "  'AIMessage': {'title': 'AIMessage',\n",
       "   'description': 'A Message from an AI.',\n",
       "   'type': 'object',\n",
       "   'properties': {'content': {'title': 'Content', 'type': 'string'},\n",
       "    'additional_kwargs': {'title': 'Additional Kwargs', 'type': 'object'},\n",
       "    'type': {'title': 'Type',\n",
       "     'default': 'ai',\n",
       "     'enum': ['ai'],\n",
       "     'type': 'string'},\n",
       "    'example': {'title': 'Example', 'default': False, 'type': 'boolean'}},\n",
       "   'required': ['content']},\n",
       "  'ChatMessage': {'title': 'ChatMessage',\n",
       "   'description': 'A Message that can be assigned an arbitrary speaker (i.e. role).',\n",
       "   'type': 'object',\n",
       "   'properties': {'content': {'title': 'Content', 'type': 'string'},\n",
       "    'additional_kwargs': {'title': 'Additional Kwargs', 'type': 'object'},\n",
       "    'type': {'title': 'Type',\n",
       "     'default': 'chat',\n",
       "     'enum': ['chat'],\n",
       "     'type': 'string'},\n",
       "    'role': {'title': 'Role', 'type': 'string'}},\n",
       "   'required': ['content', 'role']},\n",
       "  'FunctionMessage': {'title': 'FunctionMessage',\n",
       "   'description': 'A Message for passing the result of executing a function back to a model.',\n",
       "   'type': 'object',\n",
       "   'properties': {'content': {'title': 'Content', 'type': 'string'},\n",
       "    'additional_kwargs': {'title': 'Additional Kwargs', 'type': 'object'},\n",
       "    'type': {'title': 'Type',\n",
       "     'default': 'function',\n",
       "     'enum': ['function'],\n",
       "     'type': 'string'},\n",
       "    'name': {'title': 'Name', 'type': 'string'}},\n",
       "   'required': ['content', 'name']},\n",
       "  'SystemMessage': {'title': 'SystemMessage',\n",
       "   'description': 'A Message for priming AI behavior, usually passed in as the first of a sequence\\nof input messages.',\n",
       "   'type': 'object',\n",
       "   'properties': {'content': {'title': 'Content', 'type': 'string'},\n",
       "    'additional_kwargs': {'title': 'Additional Kwargs', 'type': 'object'},\n",
       "    'type': {'title': 'Type',\n",
       "     'default': 'system',\n",
       "     'enum': ['system'],\n",
       "     'type': 'string'}},\n",
       "   'required': ['content']}}}"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The output schema of the chain is the output schema of its last part, in this case a ChatModel, which outputs a ChatMessage\n",
    "chain.output_schema.schema()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "daf2b2b2",
   "metadata": {},
   "source": [
    "## Stream"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "bea9639d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Why don't bears wear shoes?\n",
      "\n",
      "Because they already have bear feet!"
     ]
    }
   ],
   "source": [
    "for s in chain.stream({\"topic\": \"bears\"}):\n",
    "    print(s.content, end=\"\", flush=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cbf1c782",
   "metadata": {},
   "source": [
    "## Invoke"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "470e483f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AIMessage(content=\"Why don't bears wear shoes?\\n\\nBecause they already have bear feet!\")"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chain.invoke({\"topic\": \"bears\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88f0c279",
   "metadata": {},
   "source": [
    "## Batch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "9685de67",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[AIMessage(content=\"Why don't bears wear shoes?\\n\\nBecause they have bear feet!\"),\n",
       " AIMessage(content=\"Why don't cats play poker in the wild?\\n\\nToo many cheetahs!\")]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chain.batch([{\"topic\": \"bears\"}, {\"topic\": \"cats\"}])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2434ab15",
   "metadata": {},
   "source": [
    "You can set the number of concurrent requests by using the `max_concurrency` parameter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "a08522f6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[AIMessage(content=\"Why don't bears wear shoes? \\n\\nBecause they have bear feet!\"),\n",
       " AIMessage(content=\"Why don't cats play poker in the wild?\\n\\nToo many cheetahs!\")]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chain.batch([{\"topic\": \"bears\"}, {\"topic\": \"cats\"}], config={\"max_concurrency\": 5})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b960cbfe",
   "metadata": {},
   "source": [
    "## Async Stream"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "ea35eee4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sure, here's a bear-themed joke for you:\n",
      "\n",
      "Why don't bears wear shoes?\n",
      "\n",
      "Because they already have bear feet!"
     ]
    }
   ],
   "source": [
    "async for s in chain.astream({\"topic\": \"bears\"}):\n",
    "    print(s.content, end=\"\", flush=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04cb3324",
   "metadata": {},
   "source": [
    "## Async Invoke"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "ef8c9b20",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AIMessage(content=\"Why don't bears wear shoes? \\n\\nBecause they have bear feet!\")"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "await chain.ainvoke({\"topic\": \"bears\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3da288d5",
   "metadata": {},
   "source": [
    "## Async Batch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "eba2a103",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[AIMessage(content=\"Why don't bears wear shoes?\\n\\nBecause they have bear feet!\")]"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "await chain.abatch([{\"topic\": \"bears\"}])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9cef104",
   "metadata": {},
   "source": [
    "## Async Stream Intermediate Steps\n",
    "\n",
    "All runnables also have a method `.astream_log()` which is used to stream (as they happen) all or part of the intermediate steps of your chain/sequence. \n",
    "\n",
    "This is useful to show progress to the user, to use intermediate results, or to debug your chain.\n",
    "\n",
    "You can stream all steps (default) or include/exclude steps by name, tags or metadata.\n",
    "\n",
    "This method yields [JSONPatch](https://jsonpatch.com) ops that when applied in the same order as received build up the RunState.\n",
    "\n",
    "```python\n",
    "class LogEntry(TypedDict):\n",
    "    id: str\n",
    "    \"\"\"ID of the sub-run.\"\"\"\n",
    "    name: str\n",
    "    \"\"\"Name of the object being run.\"\"\"\n",
    "    type: str\n",
    "    \"\"\"Type of the object being run, eg. prompt, chain, llm, etc.\"\"\"\n",
    "    tags: List[str]\n",
    "    \"\"\"List of tags for the run.\"\"\"\n",
    "    metadata: Dict[str, Any]\n",
    "    \"\"\"Key-value pairs of metadata for the run.\"\"\"\n",
    "    start_time: str\n",
    "    \"\"\"ISO-8601 timestamp of when the run started.\"\"\"\n",
    "\n",
    "    streamed_output_str: List[str]\n",
    "    \"\"\"List of LLM tokens streamed by this run, if applicable.\"\"\"\n",
    "    final_output: Optional[Any]\n",
    "    \"\"\"Final output of this run.\n",
    "    Only available after the run has finished successfully.\"\"\"\n",
    "    end_time: Optional[str]\n",
    "    \"\"\"ISO-8601 timestamp of when the run ended.\n",
    "    Only available after the run has finished.\"\"\"\n",
    "\n",
    "\n",
    "class RunState(TypedDict):\n",
    "    id: str\n",
    "    \"\"\"ID of the run.\"\"\"\n",
    "    streamed_output: List[Any]\n",
    "    \"\"\"List of output chunks streamed by Runnable.stream()\"\"\"\n",
    "    final_output: Optional[Any]\n",
    "    \"\"\"Final output of the run, usually the result of aggregating (`+`) streamed_output.\n",
    "    Only available after the run has finished successfully.\"\"\"\n",
    "\n",
    "    logs: Dict[str, LogEntry]\n",
    "    \"\"\"Map of run names to sub-runs. If filters were supplied, this list will\n",
    "    contain only the runs that matched the filters.\"\"\"\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a146a5df-25be-4fa2-a7e4-df8ebe55a35e",
   "metadata": {},
   "source": [
    "### Streaming JSONPatch chunks\n",
    "\n",
    "This is useful eg. to stream the `JSONPatch` in an HTTP server, and then apply the ops on the client to rebuild the run state there. See [LangServe](https://github.com/langchain-ai/langserve) for tooling to make it easier to build a webserver from any Runnable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "21c9019e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----------------------------------------\n",
      "RunLogPatch({'op': 'replace',\n",
      "  'path': '',\n",
      "  'value': {'final_output': None,\n",
      "            'id': 'e2f2cc72-eb63-4d20-8326-237367482efb',\n",
      "            'logs': {},\n",
      "            'streamed_output': []}})\n",
      "----------------------------------------\n",
      "RunLogPatch({'op': 'add',\n",
      "  'path': '/logs/Docs',\n",
      "  'value': {'end_time': None,\n",
      "            'final_output': None,\n",
      "            'id': '8da492cc-4492-4e74-b8b0-9e60e8693390',\n",
      "            'metadata': {},\n",
      "            'name': 'Docs',\n",
      "            'start_time': '2023-10-19T17:50:13.526',\n",
      "            'streamed_output_str': [],\n",
      "            'tags': ['map:key:context', 'FAISS'],\n",
      "            'type': 'retriever'}})\n",
      "----------------------------------------\n",
      "RunLogPatch({'op': 'add',\n",
      "  'path': '/logs/Docs/final_output',\n",
      "  'value': {'documents': [Document(page_content='harrison worked at kensho')]}},\n",
      " {'op': 'add',\n",
      "  'path': '/logs/Docs/end_time',\n",
      "  'value': '2023-10-19T17:50:13.713'})\n",
      "----------------------------------------\n",
      "RunLogPatch({'op': 'add', 'path': '/streamed_output/-', 'value': ''})\n",
      "----------------------------------------\n",
      "RunLogPatch({'op': 'add', 'path': '/streamed_output/-', 'value': 'H'})\n",
      "----------------------------------------\n",
      "RunLogPatch({'op': 'add', 'path': '/streamed_output/-', 'value': 'arrison'})\n",
      "----------------------------------------\n",
      "RunLogPatch({'op': 'add', 'path': '/streamed_output/-', 'value': ' worked'})\n",
      "----------------------------------------\n",
      "RunLogPatch({'op': 'add', 'path': '/streamed_output/-', 'value': ' at'})\n",
      "----------------------------------------\n",
      "RunLogPatch({'op': 'add', 'path': '/streamed_output/-', 'value': ' Kens'})\n",
      "----------------------------------------\n",
      "RunLogPatch({'op': 'add', 'path': '/streamed_output/-', 'value': 'ho'})\n",
      "----------------------------------------\n",
      "RunLogPatch({'op': 'add', 'path': '/streamed_output/-', 'value': '.'})\n",
      "----------------------------------------\n",
      "RunLogPatch({'op': 'add', 'path': '/streamed_output/-', 'value': ''})\n",
      "----------------------------------------\n",
      "RunLogPatch({'op': 'replace',\n",
      "  'path': '/final_output',\n",
      "  'value': {'output': 'Harrison worked at Kensho.'}})\n"
     ]
    }
   ],
   "source": [
    "from langchain.embeddings import OpenAIEmbeddings\n",
    "from langchain.schema.output_parser import StrOutputParser\n",
    "from langchain.schema.runnable import RunnablePassthrough\n",
    "from langchain.vectorstores import FAISS\n",
    "\n",
    "template = \"\"\"Answer the question based only on the following context:\n",
    "{context}\n",
    "\n",
    "Question: {question}\n",
    "\"\"\"\n",
    "prompt = ChatPromptTemplate.from_template(template)\n",
    "\n",
    "vectorstore = FAISS.from_texts(\n",
    "    [\"harrison worked at kensho\"], embedding=OpenAIEmbeddings()\n",
    ")\n",
    "retriever = vectorstore.as_retriever()\n",
    "\n",
    "retrieval_chain = (\n",
    "    {\n",
    "        \"context\": retriever.with_config(run_name=\"Docs\"),\n",
    "        \"question\": RunnablePassthrough(),\n",
    "    }\n",
    "    | prompt\n",
    "    | model\n",
    "    | StrOutputParser()\n",
    ")\n",
    "\n",
    "async for chunk in retrieval_chain.astream_log(\n",
    "    \"where did harrison work?\", include_names=[\"Docs\"]\n",
    "):\n",
    "    print(\"-\" * 40)\n",
    "    print(chunk)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19570f36-7126-4fe2-b209-0cc6178b4582",
   "metadata": {},
   "source": [
    "### Streaming the incremental RunState\n",
    "\n",
    "You can simply pass `diff=False` to get incremental values of `RunState`. \n",
    "You get more verbose output with more repetitive parts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "5c26b731-b4eb-4967-a42a-dec813249ecb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----------------------------------------------------------------------\n",
      "RunLog({'final_output': None,\n",
      " 'id': 'afe66178-d75f-4c2d-b348-b1d144239cd6',\n",
      " 'logs': {},\n",
      " 'streamed_output': []})\n",
      "----------------------------------------------------------------------\n",
      "RunLog({'final_output': None,\n",
      " 'id': 'afe66178-d75f-4c2d-b348-b1d144239cd6',\n",
      " 'logs': {'Docs': {'end_time': None,\n",
      "                   'final_output': None,\n",
      "                   'id': '88d51118-5756-4891-89c5-2f6a5e90cc26',\n",
      "                   'metadata': {},\n",
      "                   'name': 'Docs',\n",
      "                   'start_time': '2023-10-19T17:52:15.438',\n",
      "                   'streamed_output_str': [],\n",
      "                   'tags': ['map:key:context', 'FAISS'],\n",
      "                   'type': 'retriever'}},\n",
      " 'streamed_output': []})\n",
      "----------------------------------------------------------------------\n",
      "RunLog({'final_output': None,\n",
      " 'id': 'afe66178-d75f-4c2d-b348-b1d144239cd6',\n",
      " 'logs': {'Docs': {'end_time': '2023-10-19T17:52:15.738',\n",
      "                   'final_output': {'documents': [Document(page_content='harrison worked at kensho')]},\n",
      "                   'id': '88d51118-5756-4891-89c5-2f6a5e90cc26',\n",
      "                   'metadata': {},\n",
      "                   'name': 'Docs',\n",
      "                   'start_time': '2023-10-19T17:52:15.438',\n",
      "                   'streamed_output_str': [],\n",
      "                   'tags': ['map:key:context', 'FAISS'],\n",
      "                   'type': 'retriever'}},\n",
      " 'streamed_output': []})\n",
      "----------------------------------------------------------------------\n",
      "RunLog({'final_output': None,\n",
      " 'id': 'afe66178-d75f-4c2d-b348-b1d144239cd6',\n",
      " 'logs': {'Docs': {'end_time': '2023-10-19T17:52:15.738',\n",
      "                   'final_output': {'documents': [Document(page_content='harrison worked at kensho')]},\n",
      "                   'id': '88d51118-5756-4891-89c5-2f6a5e90cc26',\n",
      "                   'metadata': {},\n",
      "                   'name': 'Docs',\n",
      "                   'start_time': '2023-10-19T17:52:15.438',\n",
      "                   'streamed_output_str': [],\n",
      "                   'tags': ['map:key:context', 'FAISS'],\n",
      "                   'type': 'retriever'}},\n",
      " 'streamed_output': ['']})\n",
      "----------------------------------------------------------------------\n",
      "RunLog({'final_output': None,\n",
      " 'id': 'afe66178-d75f-4c2d-b348-b1d144239cd6',\n",
      " 'logs': {'Docs': {'end_time': '2023-10-19T17:52:15.738',\n",
      "                   'final_output': {'documents': [Document(page_content='harrison worked at kensho')]},\n",
      "                   'id': '88d51118-5756-4891-89c5-2f6a5e90cc26',\n",
      "                   'metadata': {},\n",
      "                   'name': 'Docs',\n",
      "                   'start_time': '2023-10-19T17:52:15.438',\n",
      "                   'streamed_output_str': [],\n",
      "                   'tags': ['map:key:context', 'FAISS'],\n",
      "                   'type': 'retriever'}},\n",
      " 'streamed_output': ['', 'H']})\n",
      "----------------------------------------------------------------------\n",
      "RunLog({'final_output': None,\n",
      " 'id': 'afe66178-d75f-4c2d-b348-b1d144239cd6',\n",
      " 'logs': {'Docs': {'end_time': '2023-10-19T17:52:15.738',\n",
      "                   'final_output': {'documents': [Document(page_content='harrison worked at kensho')]},\n",
      "                   'id': '88d51118-5756-4891-89c5-2f6a5e90cc26',\n",
      "                   'metadata': {},\n",
      "                   'name': 'Docs',\n",
      "                   'start_time': '2023-10-19T17:52:15.438',\n",
      "                   'streamed_output_str': [],\n",
      "                   'tags': ['map:key:context', 'FAISS'],\n",
      "                   'type': 'retriever'}},\n",
      " 'streamed_output': ['', 'H', 'arrison']})\n",
      "----------------------------------------------------------------------\n",
      "RunLog({'final_output': None,\n",
      " 'id': 'afe66178-d75f-4c2d-b348-b1d144239cd6',\n",
      " 'logs': {'Docs': {'end_time': '2023-10-19T17:52:15.738',\n",
      "                   'final_output': {'documents': [Document(page_content='harrison worked at kensho')]},\n",
      "                   'id': '88d51118-5756-4891-89c5-2f6a5e90cc26',\n",
      "                   'metadata': {},\n",
      "                   'name': 'Docs',\n",
      "                   'start_time': '2023-10-19T17:52:15.438',\n",
      "                   'streamed_output_str': [],\n",
      "                   'tags': ['map:key:context', 'FAISS'],\n",
      "                   'type': 'retriever'}},\n",
      " 'streamed_output': ['', 'H', 'arrison', ' worked']})\n",
      "----------------------------------------------------------------------\n",
      "RunLog({'final_output': None,\n",
      " 'id': 'afe66178-d75f-4c2d-b348-b1d144239cd6',\n",
      " 'logs': {'Docs': {'end_time': '2023-10-19T17:52:15.738',\n",
      "                   'final_output': {'documents': [Document(page_content='harrison worked at kensho')]},\n",
      "                   'id': '88d51118-5756-4891-89c5-2f6a5e90cc26',\n",
      "                   'metadata': {},\n",
      "                   'name': 'Docs',\n",
      "                   'start_time': '2023-10-19T17:52:15.438',\n",
      "                   'streamed_output_str': [],\n",
      "                   'tags': ['map:key:context', 'FAISS'],\n",
      "                   'type': 'retriever'}},\n",
      " 'streamed_output': ['', 'H', 'arrison', ' worked', ' at']})\n",
      "----------------------------------------------------------------------\n",
      "RunLog({'final_output': None,\n",
      " 'id': 'afe66178-d75f-4c2d-b348-b1d144239cd6',\n",
      " 'logs': {'Docs': {'end_time': '2023-10-19T17:52:15.738',\n",
      "                   'final_output': {'documents': [Document(page_content='harrison worked at kensho')]},\n",
      "                   'id': '88d51118-5756-4891-89c5-2f6a5e90cc26',\n",
      "                   'metadata': {},\n",
      "                   'name': 'Docs',\n",
      "                   'start_time': '2023-10-19T17:52:15.438',\n",
      "                   'streamed_output_str': [],\n",
      "                   'tags': ['map:key:context', 'FAISS'],\n",
      "                   'type': 'retriever'}},\n",
      " 'streamed_output': ['', 'H', 'arrison', ' worked', ' at', ' Kens']})\n",
      "----------------------------------------------------------------------\n",
      "RunLog({'final_output': None,\n",
      " 'id': 'afe66178-d75f-4c2d-b348-b1d144239cd6',\n",
      " 'logs': {'Docs': {'end_time': '2023-10-19T17:52:15.738',\n",
      "                   'final_output': {'documents': [Document(page_content='harrison worked at kensho')]},\n",
      "                   'id': '88d51118-5756-4891-89c5-2f6a5e90cc26',\n",
      "                   'metadata': {},\n",
      "                   'name': 'Docs',\n",
      "                   'start_time': '2023-10-19T17:52:15.438',\n",
      "                   'streamed_output_str': [],\n",
      "                   'tags': ['map:key:context', 'FAISS'],\n",
      "                   'type': 'retriever'}},\n",
      " 'streamed_output': ['', 'H', 'arrison', ' worked', ' at', ' Kens', 'ho']})\n",
      "----------------------------------------------------------------------\n",
      "RunLog({'final_output': None,\n",
      " 'id': 'afe66178-d75f-4c2d-b348-b1d144239cd6',\n",
      " 'logs': {'Docs': {'end_time': '2023-10-19T17:52:15.738',\n",
      "                   'final_output': {'documents': [Document(page_content='harrison worked at kensho')]},\n",
      "                   'id': '88d51118-5756-4891-89c5-2f6a5e90cc26',\n",
      "                   'metadata': {},\n",
      "                   'name': 'Docs',\n",
      "                   'start_time': '2023-10-19T17:52:15.438',\n",
      "                   'streamed_output_str': [],\n",
      "                   'tags': ['map:key:context', 'FAISS'],\n",
      "                   'type': 'retriever'}},\n",
      " 'streamed_output': ['', 'H', 'arrison', ' worked', ' at', ' Kens', 'ho', '.']})\n",
      "----------------------------------------------------------------------\n",
      "RunLog({'final_output': None,\n",
      " 'id': 'afe66178-d75f-4c2d-b348-b1d144239cd6',\n",
      " 'logs': {'Docs': {'end_time': '2023-10-19T17:52:15.738',\n",
      "                   'final_output': {'documents': [Document(page_content='harrison worked at kensho')]},\n",
      "                   'id': '88d51118-5756-4891-89c5-2f6a5e90cc26',\n",
      "                   'metadata': {},\n",
      "                   'name': 'Docs',\n",
      "                   'start_time': '2023-10-19T17:52:15.438',\n",
      "                   'streamed_output_str': [],\n",
      "                   'tags': ['map:key:context', 'FAISS'],\n",
      "                   'type': 'retriever'}},\n",
      " 'streamed_output': ['',\n",
      "                     'H',\n",
      "                     'arrison',\n",
      "                     ' worked',\n",
      "                     ' at',\n",
      "                     ' Kens',\n",
      "                     'ho',\n",
      "                     '.',\n",
      "                     '']})\n",
      "----------------------------------------------------------------------\n",
      "RunLog({'final_output': {'output': 'Harrison worked at Kensho.'},\n",
      " 'id': 'afe66178-d75f-4c2d-b348-b1d144239cd6',\n",
      " 'logs': {'Docs': {'end_time': '2023-10-19T17:52:15.738',\n",
      "                   'final_output': {'documents': [Document(page_content='harrison worked at kensho')]},\n",
      "                   'id': '88d51118-5756-4891-89c5-2f6a5e90cc26',\n",
      "                   'metadata': {},\n",
      "                   'name': 'Docs',\n",
      "                   'start_time': '2023-10-19T17:52:15.438',\n",
      "                   'streamed_output_str': [],\n",
      "                   'tags': ['map:key:context', 'FAISS'],\n",
      "                   'type': 'retriever'}},\n",
      " 'streamed_output': ['',\n",
      "                     'H',\n",
      "                     'arrison',\n",
      "                     ' worked',\n",
      "                     ' at',\n",
      "                     ' Kens',\n",
      "                     'ho',\n",
      "                     '.',\n",
      "                     '']})\n"
     ]
    }
   ],
   "source": [
    "async for chunk in retrieval_chain.astream_log(\n",
    "    \"where did harrison work?\", include_names=[\"Docs\"], diff=False\n",
    "):\n",
    "    print(\"-\" * 70)\n",
    "    print(chunk)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7006f1aa",
   "metadata": {},
   "source": [
    "## Parallelism\n",
    "\n",
    "Let's take a look at how LangChain Expression Language supports parallel requests. \n",
    "For example, when using a `RunnableParallel` (often written as a dictionary) it executes each element in parallel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "0a1c409d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.schema.runnable import RunnableParallel\n",
    "\n",
    "chain1 = ChatPromptTemplate.from_template(\"tell me a joke about {topic}\") | model\n",
    "chain2 = (\n",
    "    ChatPromptTemplate.from_template(\"write a short (2 line) poem about {topic}\")\n",
    "    | model\n",
    ")\n",
    "combined = RunnableParallel(joke=chain1, poem=chain2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "08044c0a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 54.3 ms, sys: 0 ns, total: 54.3 ms\n",
      "Wall time: 2.29 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "AIMessage(content=\"Why don't bears wear shoes?\\n\\nBecause they already have bear feet!\")"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "chain1.invoke({\"topic\": \"bears\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "22c56804",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 7.8 ms, sys: 0 ns, total: 7.8 ms\n",
      "Wall time: 1.43 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "AIMessage(content=\"In wild embrace,\\nNature's strength roams with grace.\")"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "chain2.invoke({\"topic\": \"bears\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "4fff4cbb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 167 ms, sys: 921 µs, total: 168 ms\n",
      "Wall time: 1.56 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'joke': AIMessage(content=\"Why don't bears wear shoes?\\n\\nBecause they already have bear feet!\"),\n",
       " 'poem': AIMessage(content=\"Fierce and wild, nature's might,\\nBears roam the woods, shadows of the night.\")}"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "combined.invoke({\"topic\": \"bears\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80164216-0abd-439b-8407-409539e104b6",
   "metadata": {},
   "source": [
    "### Parallelism on batches\n",
    "\n",
    "Parallelism can be combined with other runnables.\n",
    "Let's try to use parallelism with batches."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "f67d2268-c766-441b-8d64-57b8219ccb34",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 159 ms, sys: 3.66 ms, total: 163 ms\n",
      "Wall time: 1.34 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[AIMessage(content=\"Why don't bears wear shoes?\\n\\nBecause they already have bear feet!\"),\n",
       " AIMessage(content=\"Sure, here's a cat joke for you:\\n\\nWhy don't cats play poker in the wild?\\n\\nBecause there are too many cheetahs!\")]"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "chain1.batch([{\"topic\": \"bears\"}, {\"topic\": \"cats\"}])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "83c8d511-9563-403e-9c06-cae986cf5dee",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 165 ms, sys: 0 ns, total: 165 ms\n",
      "Wall time: 1.73 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[AIMessage(content=\"Silent giants roam,\\nNature's strength, love's emblem shown.\"),\n",
       " AIMessage(content='Whiskers aglow, paws tiptoe,\\nGraceful hunters, hearts aglow.')]"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "chain2.batch([{\"topic\": \"bears\"}, {\"topic\": \"cats\"}])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "07a81230-8db8-4b96-bdcb-99ae1d171f2f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 507 ms, sys: 125 ms, total: 632 ms\n",
      "Wall time: 1.49 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[{'joke': AIMessage(content=\"Why don't bears wear shoes?\\n\\nBecause they already have bear feet!\"),\n",
       "  'poem': AIMessage(content=\"Majestic bears roam,\\nNature's wild guardians of home.\")},\n",
       " {'joke': AIMessage(content=\"Sure, here's a cat joke for you:\\n\\nWhy did the cat sit on the computer?\\n\\nBecause it wanted to keep an eye on the mouse!\"),\n",
       "  'poem': AIMessage(content='Whiskers twitch, eyes gleam,\\nGraceful creatures, feline dream.')}]"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "combined.batch([{\"topic\": \"bears\"}, {\"topic\": \"cats\"}])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5218cafd-370a-4e3a-85a0-452e570092fe",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
