mirror of
https://github.com/run-llama/rag-bedrock.git
synced 2026-07-01 01:37:54 -04:00
wip
This commit is contained in:
Simon Suo
@@ -0,0 +1,2 @@
|
||||
.ipynb_checkpoints
|
||||
data
|
||||
@@ -1,507 +0,0 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "64be232e-ae9d-487d-b27a-1321554e21ba",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# 1 - Ingest with LlamaParse into S3 for KB"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "d255267b-253c-4965-998f-c38a3a168339",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"In this notebook, we use LlamaParse to pre-process complex documents and stage them in S3 for Amazon Bedrock Knowledge Base (KB)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "24ff9ba5-4187-4eb5-bc05-20a39bb3cd2b",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### LlamaParse is a document parser optimized for RAG over complex documents\n",
|
||||
"- ✅ Extracts tables / charts\n",
|
||||
"- ✅ Input natural language parsing instructions\n",
|
||||
"- ✅ JSON mode\n",
|
||||
"- ✅ Image Extraction\n",
|
||||
"- ✅ Support for ~10+ document types (.pdf, .pptx, .docx, .xml)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "445f5647-56b4-40a2-81d8-6c2e50bb4a83",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Installation"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "c173cb8e-fe1e-4c43-8a08-f93e8e733c21",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Install llama-index (core framework) and llama-parse (LlamaParse client)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 90,
|
||||
"id": "e126c2e2-2da7-4777-9a7f-43862063ba69",
|
||||
"metadata": {
|
||||
"scrolled": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Requirement already satisfied: llama-index in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (0.10.30)\n",
|
||||
"Requirement already satisfied: llama-index-agent-openai<0.3.0,>=0.1.4 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index) (0.2.2)\n",
|
||||
"Requirement already satisfied: llama-index-cli<0.2.0,>=0.1.2 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index) (0.1.12)\n",
|
||||
"Requirement already satisfied: llama-index-core<0.11.0,>=0.10.30 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index) (0.10.30)\n",
|
||||
"Requirement already satisfied: llama-index-embeddings-openai<0.2.0,>=0.1.5 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index) (0.1.7)\n",
|
||||
"Requirement already satisfied: llama-index-indices-managed-llama-cloud<0.2.0,>=0.1.2 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index) (0.1.5)\n",
|
||||
"Requirement already satisfied: llama-index-legacy<0.10.0,>=0.9.48 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index) (0.9.48)\n",
|
||||
"Requirement already satisfied: llama-index-llms-openai<0.2.0,>=0.1.13 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index) (0.1.15)\n",
|
||||
"Requirement already satisfied: llama-index-multi-modal-llms-openai<0.2.0,>=0.1.3 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index) (0.1.5)\n",
|
||||
"Requirement already satisfied: llama-index-program-openai<0.2.0,>=0.1.3 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index) (0.1.5)\n",
|
||||
"Requirement already satisfied: llama-index-question-gen-openai<0.2.0,>=0.1.2 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index) (0.1.3)\n",
|
||||
"Requirement already satisfied: llama-index-readers-file<0.2.0,>=0.1.4 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index) (0.1.19)\n",
|
||||
"Requirement already satisfied: llama-index-readers-llama-parse<0.2.0,>=0.1.2 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index) (0.1.4)\n",
|
||||
"Requirement already satisfied: openai>=1.14.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-agent-openai<0.3.0,>=0.1.4->llama-index) (1.17.0)\n",
|
||||
"Requirement already satisfied: PyYAML>=6.0.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (6.0.1)\n",
|
||||
"Requirement already satisfied: SQLAlchemy[asyncio]>=1.4.49 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (2.0.29)\n",
|
||||
"Requirement already satisfied: aiohttp<4.0.0,>=3.8.6 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (3.9.4)\n",
|
||||
"Requirement already satisfied: dataclasses-json in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (0.6.4)\n",
|
||||
"Requirement already satisfied: deprecated>=1.2.9.3 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (1.2.14)\n",
|
||||
"Requirement already satisfied: dirtyjson<2.0.0,>=1.0.8 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (1.0.8)\n",
|
||||
"Requirement already satisfied: fsspec>=2023.5.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (2024.3.1)\n",
|
||||
"Requirement already satisfied: httpx in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (0.27.0)\n",
|
||||
"Requirement already satisfied: llamaindex-py-client<0.2.0,>=0.1.18 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (0.1.18)\n",
|
||||
"Requirement already satisfied: nest-asyncio<2.0.0,>=1.5.8 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (1.6.0)\n",
|
||||
"Requirement already satisfied: networkx>=3.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (3.3)\n",
|
||||
"Requirement already satisfied: nltk<4.0.0,>=3.8.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (3.8.1)\n",
|
||||
"Requirement already satisfied: numpy in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (1.26.4)\n",
|
||||
"Requirement already satisfied: pandas in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (2.2.2)\n",
|
||||
"Requirement already satisfied: pillow>=9.0.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (10.3.0)\n",
|
||||
"Requirement already satisfied: requests>=2.31.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (2.31.0)\n",
|
||||
"Requirement already satisfied: tenacity<9.0.0,>=8.2.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (8.2.3)\n",
|
||||
"Requirement already satisfied: tiktoken>=0.3.3 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (0.6.0)\n",
|
||||
"Requirement already satisfied: tqdm<5.0.0,>=4.66.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (4.66.2)\n",
|
||||
"Requirement already satisfied: typing-extensions>=4.5.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (4.11.0)\n",
|
||||
"Requirement already satisfied: typing-inspect>=0.8.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (0.9.0)\n",
|
||||
"Requirement already satisfied: wrapt in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (1.16.0)\n",
|
||||
"Requirement already satisfied: beautifulsoup4<5.0.0,>=4.12.3 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-readers-file<0.2.0,>=0.1.4->llama-index) (4.12.3)\n",
|
||||
"Requirement already satisfied: pypdf<5.0.0,>=4.0.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-readers-file<0.2.0,>=0.1.4->llama-index) (4.2.0)\n",
|
||||
"Requirement already satisfied: striprtf<0.0.27,>=0.0.26 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-readers-file<0.2.0,>=0.1.4->llama-index) (0.0.26)\n",
|
||||
"Requirement already satisfied: llama-parse<0.5.0,>=0.4.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-readers-llama-parse<0.2.0,>=0.1.2->llama-index) (0.4.0)\n",
|
||||
"Requirement already satisfied: aiosignal>=1.1.2 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core<0.11.0,>=0.10.30->llama-index) (1.3.1)\n",
|
||||
"Requirement already satisfied: attrs>=17.3.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core<0.11.0,>=0.10.30->llama-index) (23.2.0)\n",
|
||||
"Requirement already satisfied: frozenlist>=1.1.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core<0.11.0,>=0.10.30->llama-index) (1.4.1)\n",
|
||||
"Requirement already satisfied: multidict<7.0,>=4.5 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core<0.11.0,>=0.10.30->llama-index) (6.0.5)\n",
|
||||
"Requirement already satisfied: yarl<2.0,>=1.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core<0.11.0,>=0.10.30->llama-index) (1.9.4)\n",
|
||||
"Requirement already satisfied: soupsieve>1.2 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from beautifulsoup4<5.0.0,>=4.12.3->llama-index-readers-file<0.2.0,>=0.1.4->llama-index) (2.5)\n",
|
||||
"Requirement already satisfied: pydantic>=1.10 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llamaindex-py-client<0.2.0,>=0.1.18->llama-index-core<0.11.0,>=0.10.30->llama-index) (2.7.0)\n",
|
||||
"Requirement already satisfied: anyio in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from httpx->llama-index-core<0.11.0,>=0.10.30->llama-index) (4.3.0)\n",
|
||||
"Requirement already satisfied: certifi in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from httpx->llama-index-core<0.11.0,>=0.10.30->llama-index) (2024.2.2)\n",
|
||||
"Requirement already satisfied: httpcore==1.* in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from httpx->llama-index-core<0.11.0,>=0.10.30->llama-index) (1.0.5)\n",
|
||||
"Requirement already satisfied: idna in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from httpx->llama-index-core<0.11.0,>=0.10.30->llama-index) (3.7)\n",
|
||||
"Requirement already satisfied: sniffio in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from httpx->llama-index-core<0.11.0,>=0.10.30->llama-index) (1.3.1)\n",
|
||||
"Requirement already satisfied: h11<0.15,>=0.13 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from httpcore==1.*->httpx->llama-index-core<0.11.0,>=0.10.30->llama-index) (0.14.0)\n",
|
||||
"Requirement already satisfied: click in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from nltk<4.0.0,>=3.8.1->llama-index-core<0.11.0,>=0.10.30->llama-index) (8.1.7)\n",
|
||||
"Requirement already satisfied: joblib in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from nltk<4.0.0,>=3.8.1->llama-index-core<0.11.0,>=0.10.30->llama-index) (1.4.0)\n",
|
||||
"Requirement already satisfied: regex>=2021.8.3 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from nltk<4.0.0,>=3.8.1->llama-index-core<0.11.0,>=0.10.30->llama-index) (2023.12.25)\n",
|
||||
"Requirement already satisfied: distro<2,>=1.7.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from openai>=1.14.0->llama-index-agent-openai<0.3.0,>=0.1.4->llama-index) (1.9.0)\n",
|
||||
"Requirement already satisfied: charset-normalizer<4,>=2 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from requests>=2.31.0->llama-index-core<0.11.0,>=0.10.30->llama-index) (3.3.2)\n",
|
||||
"Requirement already satisfied: urllib3<3,>=1.21.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from requests>=2.31.0->llama-index-core<0.11.0,>=0.10.30->llama-index) (1.26.18)\n",
|
||||
"Requirement already satisfied: greenlet!=0.4.17 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from SQLAlchemy[asyncio]>=1.4.49->llama-index-core<0.11.0,>=0.10.30->llama-index) (3.0.3)\n",
|
||||
"Requirement already satisfied: mypy-extensions>=0.3.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from typing-inspect>=0.8.0->llama-index-core<0.11.0,>=0.10.30->llama-index) (1.0.0)\n",
|
||||
"Requirement already satisfied: marshmallow<4.0.0,>=3.18.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from dataclasses-json->llama-index-core<0.11.0,>=0.10.30->llama-index) (3.21.1)\n",
|
||||
"Requirement already satisfied: python-dateutil>=2.8.2 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from pandas->llama-index-core<0.11.0,>=0.10.30->llama-index) (2.9.0.post0)\n",
|
||||
"Requirement already satisfied: pytz>=2020.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from pandas->llama-index-core<0.11.0,>=0.10.30->llama-index) (2024.1)\n",
|
||||
"Requirement already satisfied: tzdata>=2022.7 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from pandas->llama-index-core<0.11.0,>=0.10.30->llama-index) (2024.1)\n",
|
||||
"Requirement already satisfied: packaging>=17.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from marshmallow<4.0.0,>=3.18.0->dataclasses-json->llama-index-core<0.11.0,>=0.10.30->llama-index) (24.0)\n",
|
||||
"Requirement already satisfied: annotated-types>=0.4.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from pydantic>=1.10->llamaindex-py-client<0.2.0,>=0.1.18->llama-index-core<0.11.0,>=0.10.30->llama-index) (0.6.0)\n",
|
||||
"Requirement already satisfied: pydantic-core==2.18.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from pydantic>=1.10->llamaindex-py-client<0.2.0,>=0.1.18->llama-index-core<0.11.0,>=0.10.30->llama-index) (2.18.1)\n",
|
||||
"Requirement already satisfied: six>=1.5 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from python-dateutil>=2.8.2->pandas->llama-index-core<0.11.0,>=0.10.30->llama-index) (1.16.0)\n",
|
||||
"\n",
|
||||
"\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip available: \u001b[0m\u001b[31;49m22.3.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.0\u001b[0m\n",
|
||||
"\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n",
|
||||
"Note: you may need to restart the kernel to use updated packages.\n",
|
||||
"Requirement already satisfied: llama-parse in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (0.4.0)\n",
|
||||
"Requirement already satisfied: llama-index-core>=0.10.7 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-parse) (0.10.30)\n",
|
||||
"Requirement already satisfied: PyYAML>=6.0.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core>=0.10.7->llama-parse) (6.0.1)\n",
|
||||
"Requirement already satisfied: SQLAlchemy[asyncio]>=1.4.49 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core>=0.10.7->llama-parse) (2.0.29)\n",
|
||||
"Requirement already satisfied: aiohttp<4.0.0,>=3.8.6 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core>=0.10.7->llama-parse) (3.9.4)\n",
|
||||
"Requirement already satisfied: dataclasses-json in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core>=0.10.7->llama-parse) (0.6.4)\n",
|
||||
"Requirement already satisfied: deprecated>=1.2.9.3 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core>=0.10.7->llama-parse) (1.2.14)\n",
|
||||
"Requirement already satisfied: dirtyjson<2.0.0,>=1.0.8 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core>=0.10.7->llama-parse) (1.0.8)\n",
|
||||
"Requirement already satisfied: fsspec>=2023.5.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core>=0.10.7->llama-parse) (2024.3.1)\n",
|
||||
"Requirement already satisfied: httpx in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core>=0.10.7->llama-parse) (0.27.0)\n",
|
||||
"Requirement already satisfied: llamaindex-py-client<0.2.0,>=0.1.18 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core>=0.10.7->llama-parse) (0.1.18)\n",
|
||||
"Requirement already satisfied: nest-asyncio<2.0.0,>=1.5.8 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core>=0.10.7->llama-parse) (1.6.0)\n",
|
||||
"Requirement already satisfied: networkx>=3.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core>=0.10.7->llama-parse) (3.3)\n",
|
||||
"Requirement already satisfied: nltk<4.0.0,>=3.8.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core>=0.10.7->llama-parse) (3.8.1)\n",
|
||||
"Requirement already satisfied: numpy in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core>=0.10.7->llama-parse) (1.26.4)\n",
|
||||
"Requirement already satisfied: openai>=1.1.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core>=0.10.7->llama-parse) (1.17.0)\n",
|
||||
"Requirement already satisfied: pandas in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core>=0.10.7->llama-parse) (2.2.2)\n",
|
||||
"Requirement already satisfied: pillow>=9.0.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core>=0.10.7->llama-parse) (10.3.0)\n",
|
||||
"Requirement already satisfied: requests>=2.31.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core>=0.10.7->llama-parse) (2.31.0)\n",
|
||||
"Requirement already satisfied: tenacity<9.0.0,>=8.2.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core>=0.10.7->llama-parse) (8.2.3)\n",
|
||||
"Requirement already satisfied: tiktoken>=0.3.3 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core>=0.10.7->llama-parse) (0.6.0)\n",
|
||||
"Requirement already satisfied: tqdm<5.0.0,>=4.66.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core>=0.10.7->llama-parse) (4.66.2)\n",
|
||||
"Requirement already satisfied: typing-extensions>=4.5.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core>=0.10.7->llama-parse) (4.11.0)\n",
|
||||
"Requirement already satisfied: typing-inspect>=0.8.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core>=0.10.7->llama-parse) (0.9.0)\n",
|
||||
"Requirement already satisfied: wrapt in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core>=0.10.7->llama-parse) (1.16.0)\n",
|
||||
"Requirement already satisfied: aiosignal>=1.1.2 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core>=0.10.7->llama-parse) (1.3.1)\n",
|
||||
"Requirement already satisfied: attrs>=17.3.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core>=0.10.7->llama-parse) (23.2.0)\n",
|
||||
"Requirement already satisfied: frozenlist>=1.1.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core>=0.10.7->llama-parse) (1.4.1)\n",
|
||||
"Requirement already satisfied: multidict<7.0,>=4.5 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core>=0.10.7->llama-parse) (6.0.5)\n",
|
||||
"Requirement already satisfied: yarl<2.0,>=1.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core>=0.10.7->llama-parse) (1.9.4)\n",
|
||||
"Requirement already satisfied: pydantic>=1.10 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llamaindex-py-client<0.2.0,>=0.1.18->llama-index-core>=0.10.7->llama-parse) (2.7.0)\n",
|
||||
"Requirement already satisfied: anyio in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from httpx->llama-index-core>=0.10.7->llama-parse) (4.3.0)\n",
|
||||
"Requirement already satisfied: certifi in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from httpx->llama-index-core>=0.10.7->llama-parse) (2024.2.2)\n",
|
||||
"Requirement already satisfied: httpcore==1.* in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from httpx->llama-index-core>=0.10.7->llama-parse) (1.0.5)\n",
|
||||
"Requirement already satisfied: idna in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from httpx->llama-index-core>=0.10.7->llama-parse) (3.7)\n",
|
||||
"Requirement already satisfied: sniffio in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from httpx->llama-index-core>=0.10.7->llama-parse) (1.3.1)\n",
|
||||
"Requirement already satisfied: h11<0.15,>=0.13 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from httpcore==1.*->httpx->llama-index-core>=0.10.7->llama-parse) (0.14.0)\n",
|
||||
"Requirement already satisfied: click in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from nltk<4.0.0,>=3.8.1->llama-index-core>=0.10.7->llama-parse) (8.1.7)\n",
|
||||
"Requirement already satisfied: joblib in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from nltk<4.0.0,>=3.8.1->llama-index-core>=0.10.7->llama-parse) (1.4.0)\n",
|
||||
"Requirement already satisfied: regex>=2021.8.3 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from nltk<4.0.0,>=3.8.1->llama-index-core>=0.10.7->llama-parse) (2023.12.25)\n",
|
||||
"Requirement already satisfied: distro<2,>=1.7.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from openai>=1.1.0->llama-index-core>=0.10.7->llama-parse) (1.9.0)\n",
|
||||
"Requirement already satisfied: charset-normalizer<4,>=2 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from requests>=2.31.0->llama-index-core>=0.10.7->llama-parse) (3.3.2)\n",
|
||||
"Requirement already satisfied: urllib3<3,>=1.21.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from requests>=2.31.0->llama-index-core>=0.10.7->llama-parse) (1.26.18)\n",
|
||||
"Requirement already satisfied: greenlet!=0.4.17 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from SQLAlchemy[asyncio]>=1.4.49->llama-index-core>=0.10.7->llama-parse) (3.0.3)\n",
|
||||
"Requirement already satisfied: mypy-extensions>=0.3.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from typing-inspect>=0.8.0->llama-index-core>=0.10.7->llama-parse) (1.0.0)\n",
|
||||
"Requirement already satisfied: marshmallow<4.0.0,>=3.18.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from dataclasses-json->llama-index-core>=0.10.7->llama-parse) (3.21.1)\n",
|
||||
"Requirement already satisfied: python-dateutil>=2.8.2 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from pandas->llama-index-core>=0.10.7->llama-parse) (2.9.0.post0)\n",
|
||||
"Requirement already satisfied: pytz>=2020.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from pandas->llama-index-core>=0.10.7->llama-parse) (2024.1)\n",
|
||||
"Requirement already satisfied: tzdata>=2022.7 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from pandas->llama-index-core>=0.10.7->llama-parse) (2024.1)\n",
|
||||
"Requirement already satisfied: packaging>=17.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from marshmallow<4.0.0,>=3.18.0->dataclasses-json->llama-index-core>=0.10.7->llama-parse) (24.0)\n",
|
||||
"Requirement already satisfied: annotated-types>=0.4.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from pydantic>=1.10->llamaindex-py-client<0.2.0,>=0.1.18->llama-index-core>=0.10.7->llama-parse) (0.6.0)\n",
|
||||
"Requirement already satisfied: pydantic-core==2.18.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from pydantic>=1.10->llamaindex-py-client<0.2.0,>=0.1.18->llama-index-core>=0.10.7->llama-parse) (2.18.1)\n",
|
||||
"Requirement already satisfied: six>=1.5 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from python-dateutil>=2.8.2->pandas->llama-index-core>=0.10.7->llama-parse) (1.16.0)\n",
|
||||
"\n",
|
||||
"\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip available: \u001b[0m\u001b[31;49m22.3.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.0\u001b[0m\n",
|
||||
"\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n",
|
||||
"Note: you may need to restart the kernel to use updated packages.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"%pip install llama-index\n",
|
||||
"%pip install llama-parse"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "6f3ba3bc-78b8-46c5-8964-725a46b4d2d9",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Setup and imports"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"id": "b1ce7212-1ec3-472f-bfe6-20f2d515e66b",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import nest_asyncio\n",
|
||||
"\n",
|
||||
"nest_asyncio.apply()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "a195eac4-081d-49a7-9bde-b60169e222d0",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Get API key from http://cloud.llamaindex.ai/ and configure via env variable"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"id": "f796a6f0-83f2-42ee-b6e4-542f28821657",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"\n",
|
||||
"os.environ['LLAMA_CLOUD_API_KEY'] = 'llx-2VStQ0IBwnbc7v5xTvhE1eOh0n6Vd2NDU5TOyxXYv3or2yHO'"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "6294c909-dfd2-4398-8ac5-7fdeb6dfbedd",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Download data"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cc0d4ac0-2a16-4b2b-8348-4b39cd76fd7b",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"For this demo, we will build a simple knowledge base with 2 10K filings for uber and lyft."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 62,
|
||||
"id": "02ab751f-2e88-4be8-9aff-bf84c23098c3",
|
||||
"metadata": {
|
||||
"scrolled": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"--2024-04-18 13:31:02-- https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/10k/lyft_2021.pdf\n",
|
||||
"Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 2606:50c0:8001::154, 2606:50c0:8002::154, 2606:50c0:8003::154, ...\n",
|
||||
"Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|2606:50c0:8001::154|:443... connected.\n",
|
||||
"HTTP request sent, awaiting response... 200 OK\n",
|
||||
"Length: 1440303 (1.4M) [application/octet-stream]\n",
|
||||
"Saving to: ‘./data/10k/lyft_2021.pdf’\n",
|
||||
"\n",
|
||||
"./data/10k/lyft_202 100%[===================>] 1.37M --.-KB/s in 0.07s \n",
|
||||
"\n",
|
||||
"2024-04-18 13:31:02 (18.6 MB/s) - ‘./data/10k/lyft_2021.pdf’ saved [1440303/1440303]\n",
|
||||
"\n",
|
||||
"--2024-04-18 13:31:02-- https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/10k/uber_2021.pdf\n",
|
||||
"Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 2606:50c0:8001::154, 2606:50c0:8002::154, 2606:50c0:8003::154, ...\n",
|
||||
"Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|2606:50c0:8001::154|:443... connected.\n",
|
||||
"HTTP request sent, awaiting response... 200 OK\n",
|
||||
"Length: 1880483 (1.8M) [application/octet-stream]\n",
|
||||
"Saving to: ‘./data/10k/uber_2021.pdf’\n",
|
||||
"\n",
|
||||
"./data/10k/uber_202 100%[===================>] 1.79M --.-KB/s in 0.1s \n",
|
||||
"\n",
|
||||
"2024-04-18 13:31:02 (12.4 MB/s) - ‘./data/10k/uber_2021.pdf’ saved [1880483/1880483]\n",
|
||||
"\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"!mkdir -p './data/10k/'\n",
|
||||
"!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/10k/lyft_2021.pdf' -O './data/10k/lyft_2021.pdf'\n",
|
||||
"!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/10k/uber_2021.pdf' -O './data/10k/uber_2021.pdf'"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "e1da7ba1-215d-401a-8cd1-32442bfb64f9",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Load and parse data with LlamaParse"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"id": "589e2fc0-7ffc-4de3-a297-275f476e8126",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from llama_parse import LlamaParse\n",
|
||||
"from llama_index.core import SimpleDirectoryReader\n",
|
||||
"\n",
|
||||
"parser = LlamaParse(\n",
|
||||
" api_key=os.environ.get('LLAMA_CLOUD_API_KEY'), # set via api_key param or in your env as LLAMA_CLOUD_API_KEY\n",
|
||||
" result_type=\"markdown\", # \"markdown\" and \"text\" are available\n",
|
||||
" num_workers=4, # if multiple files passed, split in `num_workers` API calls\n",
|
||||
" verbose=True,\n",
|
||||
" language=\"en\", # Optionally you can define a language, default=en\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"file_extractor = {\".pdf\": parser}\n",
|
||||
"reader = SimpleDirectoryReader(\n",
|
||||
" input_dir='data/10k/',\n",
|
||||
" file_extractor=file_extractor\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"id": "75a3d4f9-b6ad-4128-bb63-72e49bc349d3",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Started parsing the file under job_id 705de5a7-2782-40e8-a5cf-813bb198a9fe\n",
|
||||
"Started parsing the file under job_id f35906d9-4b66-4d31-99d8-6cdb087c8934\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"documents = reader.load_data()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"id": "86231a57-5595-4cf7-8fda-0bda763305f9",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"{'file_path': '/Users/suo/dev/rag-bedrock/data/10k/lyft_2021.pdf',\n",
|
||||
" 'file_name': 'lyft_2021.pdf',\n",
|
||||
" 'file_type': 'application/pdf',\n",
|
||||
" 'file_size': 1440303,\n",
|
||||
" 'creation_date': '2024-04-18',\n",
|
||||
" 'last_modified_date': '2024-04-18'}"
|
||||
]
|
||||
},
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"documents[0].metadata"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "ae1ff75a-8bea-4457-901d-025504c08c32",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Upload data as markdown and metadata file "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "eafe0255-9139-4338-93ec-441c9dd3248a",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Now, upload and stage the parsed result markdown files for ingestion in to Amazon Bedrock KB.\n",
|
||||
"\n",
|
||||
"We also create metadata json files (the format is specific to Amazong Bedrock KB)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"id": "9f7b34db-36f7-432f-ba88-3bb4baf292ab",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import boto3 \n",
|
||||
"import botocore\n",
|
||||
"\n",
|
||||
"# Create an S3 client\n",
|
||||
"s3 = boto3.client('s3')\n",
|
||||
"\n",
|
||||
"# Specify the bucket\n",
|
||||
"bucket_name = 'bedrock-kb-10ks'"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"id": "d833bf9b-8834-4f12-bc2d-840e7788c8b3",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import json\n",
|
||||
"\n",
|
||||
"def create_bucket(bucket_name):\n",
|
||||
" try:\n",
|
||||
" s3.head_bucket(Bucket=bucket_name)\n",
|
||||
" print(f\"Bucket '{bucket_name}' already exists.\")\n",
|
||||
" except botocore.exceptions.ClientError as e:\n",
|
||||
" if e.response['Error']['Code'] == '404':\n",
|
||||
" s3.create_bucket(Bucket=bucket_name)\n",
|
||||
" print(f\"Bucket '{bucket_name}' created successfully.\")\n",
|
||||
" else:\n",
|
||||
" print(f\"Error creating bucket: {str(e)}\")\n",
|
||||
" raise \n",
|
||||
"\n",
|
||||
"def upload_document(document, bucket_name):\n",
|
||||
" try:\n",
|
||||
" object_key = document.metadata['file_path']\n",
|
||||
" \n",
|
||||
" # Upload the text as a markdown file\n",
|
||||
" s3.put_object(\n",
|
||||
" Body=document.text.encode('utf-8'),\n",
|
||||
" Bucket=bucket_name,\n",
|
||||
" Key=f\"{object_key}.md\",\n",
|
||||
" ContentType='text/markdown'\n",
|
||||
" )\n",
|
||||
" print(f\"Text uploaded to S3 as '{object_key}.md'\")\n",
|
||||
"\n",
|
||||
" # Format the metadata in the desired structure\n",
|
||||
" formatted_metadata = {\n",
|
||||
" \"metadataAttributes\": document.metadata\n",
|
||||
" }\n",
|
||||
" \n",
|
||||
" # Upload the metadata as a JSON file\n",
|
||||
" metadata_json = json.dumps(formatted_metadata, indent=4)\n",
|
||||
" s3.put_object(\n",
|
||||
" Body=metadata_json.encode('utf-8'),\n",
|
||||
" Bucket=bucket_name,\n",
|
||||
" Key=f\"{object_key}.md.metadata.json\",\n",
|
||||
" ContentType='application/json'\n",
|
||||
" )\n",
|
||||
" print(f\"Metadata uploaded to S3 as '{object_key}.md.metadata.json'\")\n",
|
||||
" \n",
|
||||
" except Exception as e:\n",
|
||||
" print(f\"Error uploading document: {str(e)}\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"id": "ce181b28-3318-42d7-b8ce-15a6c659bee4",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Text uploaded to S3 as '/Users/suo/dev/rag-bedrock/data/10k/lyft_2021.pdf.md'\n",
|
||||
"Metadata uploaded to S3 as '/Users/suo/dev/rag-bedrock/data/10k/lyft_2021.pdf.md.metadata.json'\n",
|
||||
"Text uploaded to S3 as '/Users/suo/dev/rag-bedrock/data/10k/uber_2021.pdf.md'\n",
|
||||
"Metadata uploaded to S3 as '/Users/suo/dev/rag-bedrock/data/10k/uber_2021.pdf.md.metadata.json'\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"for doc in documents:\n",
|
||||
" upload_document(doc, bucket_name) "
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.11.3"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@@ -1,706 +0,0 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "6d688c05-4842-46fc-97a5-5ee2882a0aa7",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# 2 - Agentic RAG with Bedrock KB and LlamaIndex SubQuestionQueryEngine"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "ab882819-a8a0-4748-8840-e39955c8f96f",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Installation"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 32,
|
||||
"id": "3b9d72e0-91ad-403a-aea7-d3812244678b",
|
||||
"metadata": {
|
||||
"scrolled": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Requirement already satisfied: llama-index in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (0.10.30)\n",
|
||||
"Requirement already satisfied: llama-index-agent-openai<0.3.0,>=0.1.4 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index) (0.2.2)\n",
|
||||
"Requirement already satisfied: llama-index-cli<0.2.0,>=0.1.2 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index) (0.1.12)\n",
|
||||
"Requirement already satisfied: llama-index-core<0.11.0,>=0.10.30 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index) (0.10.30)\n",
|
||||
"Requirement already satisfied: llama-index-embeddings-openai<0.2.0,>=0.1.5 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index) (0.1.7)\n",
|
||||
"Requirement already satisfied: llama-index-indices-managed-llama-cloud<0.2.0,>=0.1.2 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index) (0.1.5)\n",
|
||||
"Requirement already satisfied: llama-index-legacy<0.10.0,>=0.9.48 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index) (0.9.48)\n",
|
||||
"Requirement already satisfied: llama-index-llms-openai<0.2.0,>=0.1.13 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index) (0.1.15)\n",
|
||||
"Requirement already satisfied: llama-index-multi-modal-llms-openai<0.2.0,>=0.1.3 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index) (0.1.5)\n",
|
||||
"Requirement already satisfied: llama-index-program-openai<0.2.0,>=0.1.3 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index) (0.1.5)\n",
|
||||
"Requirement already satisfied: llama-index-question-gen-openai<0.2.0,>=0.1.2 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index) (0.1.3)\n",
|
||||
"Requirement already satisfied: llama-index-readers-file<0.2.0,>=0.1.4 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index) (0.1.19)\n",
|
||||
"Requirement already satisfied: llama-index-readers-llama-parse<0.2.0,>=0.1.2 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index) (0.1.4)\n",
|
||||
"Requirement already satisfied: openai>=1.14.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-agent-openai<0.3.0,>=0.1.4->llama-index) (1.17.0)\n",
|
||||
"Requirement already satisfied: PyYAML>=6.0.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (6.0.1)\n",
|
||||
"Requirement already satisfied: SQLAlchemy[asyncio]>=1.4.49 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (2.0.29)\n",
|
||||
"Requirement already satisfied: aiohttp<4.0.0,>=3.8.6 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (3.9.4)\n",
|
||||
"Requirement already satisfied: dataclasses-json in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (0.6.4)\n",
|
||||
"Requirement already satisfied: deprecated>=1.2.9.3 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (1.2.14)\n",
|
||||
"Requirement already satisfied: dirtyjson<2.0.0,>=1.0.8 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (1.0.8)\n",
|
||||
"Requirement already satisfied: fsspec>=2023.5.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (2024.3.1)\n",
|
||||
"Requirement already satisfied: httpx in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (0.27.0)\n",
|
||||
"Requirement already satisfied: llamaindex-py-client<0.2.0,>=0.1.18 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (0.1.18)\n",
|
||||
"Requirement already satisfied: nest-asyncio<2.0.0,>=1.5.8 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (1.6.0)\n",
|
||||
"Requirement already satisfied: networkx>=3.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (3.3)\n",
|
||||
"Requirement already satisfied: nltk<4.0.0,>=3.8.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (3.8.1)\n",
|
||||
"Requirement already satisfied: numpy in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (1.26.4)\n",
|
||||
"Requirement already satisfied: pandas in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (2.2.2)\n",
|
||||
"Requirement already satisfied: pillow>=9.0.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (10.3.0)\n",
|
||||
"Requirement already satisfied: requests>=2.31.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (2.31.0)\n",
|
||||
"Requirement already satisfied: tenacity<9.0.0,>=8.2.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (8.2.3)\n",
|
||||
"Requirement already satisfied: tiktoken>=0.3.3 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (0.6.0)\n",
|
||||
"Requirement already satisfied: tqdm<5.0.0,>=4.66.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (4.66.2)\n",
|
||||
"Requirement already satisfied: typing-extensions>=4.5.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (4.11.0)\n",
|
||||
"Requirement already satisfied: typing-inspect>=0.8.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (0.9.0)\n",
|
||||
"Requirement already satisfied: wrapt in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (1.16.0)\n",
|
||||
"Requirement already satisfied: beautifulsoup4<5.0.0,>=4.12.3 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-readers-file<0.2.0,>=0.1.4->llama-index) (4.12.3)\n",
|
||||
"Requirement already satisfied: pypdf<5.0.0,>=4.0.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-readers-file<0.2.0,>=0.1.4->llama-index) (4.2.0)\n",
|
||||
"Requirement already satisfied: striprtf<0.0.27,>=0.0.26 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-readers-file<0.2.0,>=0.1.4->llama-index) (0.0.26)\n",
|
||||
"Requirement already satisfied: llama-parse<0.5.0,>=0.4.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-readers-llama-parse<0.2.0,>=0.1.2->llama-index) (0.4.0)\n",
|
||||
"Requirement already satisfied: aiosignal>=1.1.2 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core<0.11.0,>=0.10.30->llama-index) (1.3.1)\n",
|
||||
"Requirement already satisfied: attrs>=17.3.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core<0.11.0,>=0.10.30->llama-index) (23.2.0)\n",
|
||||
"Requirement already satisfied: frozenlist>=1.1.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core<0.11.0,>=0.10.30->llama-index) (1.4.1)\n",
|
||||
"Requirement already satisfied: multidict<7.0,>=4.5 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core<0.11.0,>=0.10.30->llama-index) (6.0.5)\n",
|
||||
"Requirement already satisfied: yarl<2.0,>=1.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core<0.11.0,>=0.10.30->llama-index) (1.9.4)\n",
|
||||
"Requirement already satisfied: soupsieve>1.2 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from beautifulsoup4<5.0.0,>=4.12.3->llama-index-readers-file<0.2.0,>=0.1.4->llama-index) (2.5)\n",
|
||||
"Requirement already satisfied: pydantic>=1.10 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llamaindex-py-client<0.2.0,>=0.1.18->llama-index-core<0.11.0,>=0.10.30->llama-index) (2.7.0)\n",
|
||||
"Requirement already satisfied: anyio in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from httpx->llama-index-core<0.11.0,>=0.10.30->llama-index) (4.3.0)\n",
|
||||
"Requirement already satisfied: certifi in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from httpx->llama-index-core<0.11.0,>=0.10.30->llama-index) (2024.2.2)\n",
|
||||
"Requirement already satisfied: httpcore==1.* in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from httpx->llama-index-core<0.11.0,>=0.10.30->llama-index) (1.0.5)\n",
|
||||
"Requirement already satisfied: idna in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from httpx->llama-index-core<0.11.0,>=0.10.30->llama-index) (3.7)\n",
|
||||
"Requirement already satisfied: sniffio in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from httpx->llama-index-core<0.11.0,>=0.10.30->llama-index) (1.3.1)\n",
|
||||
"Requirement already satisfied: h11<0.15,>=0.13 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from httpcore==1.*->httpx->llama-index-core<0.11.0,>=0.10.30->llama-index) (0.14.0)\n",
|
||||
"Requirement already satisfied: click in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from nltk<4.0.0,>=3.8.1->llama-index-core<0.11.0,>=0.10.30->llama-index) (8.1.7)\n",
|
||||
"Requirement already satisfied: joblib in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from nltk<4.0.0,>=3.8.1->llama-index-core<0.11.0,>=0.10.30->llama-index) (1.4.0)\n",
|
||||
"Requirement already satisfied: regex>=2021.8.3 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from nltk<4.0.0,>=3.8.1->llama-index-core<0.11.0,>=0.10.30->llama-index) (2023.12.25)\n",
|
||||
"Requirement already satisfied: distro<2,>=1.7.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from openai>=1.14.0->llama-index-agent-openai<0.3.0,>=0.1.4->llama-index) (1.9.0)\n",
|
||||
"Requirement already satisfied: charset-normalizer<4,>=2 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from requests>=2.31.0->llama-index-core<0.11.0,>=0.10.30->llama-index) (3.3.2)\n",
|
||||
"Requirement already satisfied: urllib3<3,>=1.21.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from requests>=2.31.0->llama-index-core<0.11.0,>=0.10.30->llama-index) (1.26.18)\n",
|
||||
"Requirement already satisfied: greenlet!=0.4.17 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from SQLAlchemy[asyncio]>=1.4.49->llama-index-core<0.11.0,>=0.10.30->llama-index) (3.0.3)\n",
|
||||
"Requirement already satisfied: mypy-extensions>=0.3.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from typing-inspect>=0.8.0->llama-index-core<0.11.0,>=0.10.30->llama-index) (1.0.0)\n",
|
||||
"Requirement already satisfied: marshmallow<4.0.0,>=3.18.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from dataclasses-json->llama-index-core<0.11.0,>=0.10.30->llama-index) (3.21.1)\n",
|
||||
"Requirement already satisfied: python-dateutil>=2.8.2 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from pandas->llama-index-core<0.11.0,>=0.10.30->llama-index) (2.9.0.post0)\n",
|
||||
"Requirement already satisfied: pytz>=2020.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from pandas->llama-index-core<0.11.0,>=0.10.30->llama-index) (2024.1)\n",
|
||||
"Requirement already satisfied: tzdata>=2022.7 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from pandas->llama-index-core<0.11.0,>=0.10.30->llama-index) (2024.1)\n",
|
||||
"Requirement already satisfied: packaging>=17.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from marshmallow<4.0.0,>=3.18.0->dataclasses-json->llama-index-core<0.11.0,>=0.10.30->llama-index) (24.0)\n",
|
||||
"Requirement already satisfied: annotated-types>=0.4.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from pydantic>=1.10->llamaindex-py-client<0.2.0,>=0.1.18->llama-index-core<0.11.0,>=0.10.30->llama-index) (0.6.0)\n",
|
||||
"Requirement already satisfied: pydantic-core==2.18.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from pydantic>=1.10->llamaindex-py-client<0.2.0,>=0.1.18->llama-index-core<0.11.0,>=0.10.30->llama-index) (2.18.1)\n",
|
||||
"Requirement already satisfied: six>=1.5 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from python-dateutil>=2.8.2->pandas->llama-index-core<0.11.0,>=0.10.30->llama-index) (1.16.0)\n",
|
||||
"\n",
|
||||
"\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip available: \u001b[0m\u001b[31;49m22.3.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.0\u001b[0m\n",
|
||||
"\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n",
|
||||
"Note: you may need to restart the kernel to use updated packages.\n",
|
||||
"Requirement already satisfied: llama-index-llms-bedrock in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (0.1.6)\n",
|
||||
"Requirement already satisfied: boto3<2.0.0,>=1.34.26 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-llms-bedrock) (1.34.85)\n",
|
||||
"Requirement already satisfied: llama-index-core<0.11.0,>=0.10.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-llms-bedrock) (0.10.30)\n",
|
||||
"Requirement already satisfied: llama-index-llms-anthropic<0.2.0,>=0.1.7 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-llms-bedrock) (0.1.10)\n",
|
||||
"Requirement already satisfied: botocore<1.35.0,>=1.34.85 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from boto3<2.0.0,>=1.34.26->llama-index-llms-bedrock) (1.34.85)\n",
|
||||
"Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from boto3<2.0.0,>=1.34.26->llama-index-llms-bedrock) (1.0.1)\n",
|
||||
"Requirement already satisfied: s3transfer<0.11.0,>=0.10.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from boto3<2.0.0,>=1.34.26->llama-index-llms-bedrock) (0.10.1)\n",
|
||||
"Requirement already satisfied: PyYAML>=6.0.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-llms-bedrock) (6.0.1)\n",
|
||||
"Requirement already satisfied: SQLAlchemy[asyncio]>=1.4.49 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-llms-bedrock) (2.0.29)\n",
|
||||
"Requirement already satisfied: aiohttp<4.0.0,>=3.8.6 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-llms-bedrock) (3.9.4)\n",
|
||||
"Requirement already satisfied: dataclasses-json in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-llms-bedrock) (0.6.4)\n",
|
||||
"Requirement already satisfied: deprecated>=1.2.9.3 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-llms-bedrock) (1.2.14)\n",
|
||||
"Requirement already satisfied: dirtyjson<2.0.0,>=1.0.8 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-llms-bedrock) (1.0.8)\n",
|
||||
"Requirement already satisfied: fsspec>=2023.5.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-llms-bedrock) (2024.3.1)\n",
|
||||
"Requirement already satisfied: httpx in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-llms-bedrock) (0.27.0)\n",
|
||||
"Requirement already satisfied: llamaindex-py-client<0.2.0,>=0.1.18 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-llms-bedrock) (0.1.18)\n",
|
||||
"Requirement already satisfied: nest-asyncio<2.0.0,>=1.5.8 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-llms-bedrock) (1.6.0)\n",
|
||||
"Requirement already satisfied: networkx>=3.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-llms-bedrock) (3.3)\n",
|
||||
"Requirement already satisfied: nltk<4.0.0,>=3.8.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-llms-bedrock) (3.8.1)\n",
|
||||
"Requirement already satisfied: numpy in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-llms-bedrock) (1.26.4)\n",
|
||||
"Requirement already satisfied: openai>=1.1.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-llms-bedrock) (1.17.0)\n",
|
||||
"Requirement already satisfied: pandas in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-llms-bedrock) (2.2.2)\n",
|
||||
"Requirement already satisfied: pillow>=9.0.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-llms-bedrock) (10.3.0)\n",
|
||||
"Requirement already satisfied: requests>=2.31.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-llms-bedrock) (2.31.0)\n",
|
||||
"Requirement already satisfied: tenacity<9.0.0,>=8.2.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-llms-bedrock) (8.2.3)\n",
|
||||
"Requirement already satisfied: tiktoken>=0.3.3 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-llms-bedrock) (0.6.0)\n",
|
||||
"Requirement already satisfied: tqdm<5.0.0,>=4.66.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-llms-bedrock) (4.66.2)\n",
|
||||
"Requirement already satisfied: typing-extensions>=4.5.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-llms-bedrock) (4.11.0)\n",
|
||||
"Requirement already satisfied: typing-inspect>=0.8.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-llms-bedrock) (0.9.0)\n",
|
||||
"Requirement already satisfied: wrapt in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-llms-bedrock) (1.16.0)\n",
|
||||
"Requirement already satisfied: anthropic<0.24.0,>=0.23.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-llms-anthropic<0.2.0,>=0.1.7->llama-index-llms-bedrock) (0.23.1)\n",
|
||||
"Requirement already satisfied: aiosignal>=1.1.2 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core<0.11.0,>=0.10.1->llama-index-llms-bedrock) (1.3.1)\n",
|
||||
"Requirement already satisfied: attrs>=17.3.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core<0.11.0,>=0.10.1->llama-index-llms-bedrock) (23.2.0)\n",
|
||||
"Requirement already satisfied: frozenlist>=1.1.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core<0.11.0,>=0.10.1->llama-index-llms-bedrock) (1.4.1)\n",
|
||||
"Requirement already satisfied: multidict<7.0,>=4.5 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core<0.11.0,>=0.10.1->llama-index-llms-bedrock) (6.0.5)\n",
|
||||
"Requirement already satisfied: yarl<2.0,>=1.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core<0.11.0,>=0.10.1->llama-index-llms-bedrock) (1.9.4)\n",
|
||||
"Requirement already satisfied: anyio<5,>=3.5.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from anthropic<0.24.0,>=0.23.1->llama-index-llms-anthropic<0.2.0,>=0.1.7->llama-index-llms-bedrock) (4.3.0)\n",
|
||||
"Requirement already satisfied: distro<2,>=1.7.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from anthropic<0.24.0,>=0.23.1->llama-index-llms-anthropic<0.2.0,>=0.1.7->llama-index-llms-bedrock) (1.9.0)\n",
|
||||
"Requirement already satisfied: pydantic<3,>=1.9.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from anthropic<0.24.0,>=0.23.1->llama-index-llms-anthropic<0.2.0,>=0.1.7->llama-index-llms-bedrock) (2.7.0)\n",
|
||||
"Requirement already satisfied: sniffio in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from anthropic<0.24.0,>=0.23.1->llama-index-llms-anthropic<0.2.0,>=0.1.7->llama-index-llms-bedrock) (1.3.1)\n",
|
||||
"Requirement already satisfied: tokenizers>=0.13.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from anthropic<0.24.0,>=0.23.1->llama-index-llms-anthropic<0.2.0,>=0.1.7->llama-index-llms-bedrock) (0.19.0)\n",
|
||||
"Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from botocore<1.35.0,>=1.34.85->boto3<2.0.0,>=1.34.26->llama-index-llms-bedrock) (2.9.0.post0)\n",
|
||||
"Requirement already satisfied: urllib3!=2.2.0,<3,>=1.25.4 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from botocore<1.35.0,>=1.34.85->boto3<2.0.0,>=1.34.26->llama-index-llms-bedrock) (1.26.18)\n",
|
||||
"Requirement already satisfied: certifi in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from httpx->llama-index-core<0.11.0,>=0.10.1->llama-index-llms-bedrock) (2024.2.2)\n",
|
||||
"Requirement already satisfied: httpcore==1.* in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from httpx->llama-index-core<0.11.0,>=0.10.1->llama-index-llms-bedrock) (1.0.5)\n",
|
||||
"Requirement already satisfied: idna in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from httpx->llama-index-core<0.11.0,>=0.10.1->llama-index-llms-bedrock) (3.7)\n",
|
||||
"Requirement already satisfied: h11<0.15,>=0.13 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from httpcore==1.*->httpx->llama-index-core<0.11.0,>=0.10.1->llama-index-llms-bedrock) (0.14.0)\n",
|
||||
"Requirement already satisfied: click in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from nltk<4.0.0,>=3.8.1->llama-index-core<0.11.0,>=0.10.1->llama-index-llms-bedrock) (8.1.7)\n",
|
||||
"Requirement already satisfied: joblib in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from nltk<4.0.0,>=3.8.1->llama-index-core<0.11.0,>=0.10.1->llama-index-llms-bedrock) (1.4.0)\n",
|
||||
"Requirement already satisfied: regex>=2021.8.3 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from nltk<4.0.0,>=3.8.1->llama-index-core<0.11.0,>=0.10.1->llama-index-llms-bedrock) (2023.12.25)\n",
|
||||
"Requirement already satisfied: charset-normalizer<4,>=2 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from requests>=2.31.0->llama-index-core<0.11.0,>=0.10.1->llama-index-llms-bedrock) (3.3.2)\n",
|
||||
"Requirement already satisfied: greenlet!=0.4.17 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from SQLAlchemy[asyncio]>=1.4.49->llama-index-core<0.11.0,>=0.10.1->llama-index-llms-bedrock) (3.0.3)\n",
|
||||
"Requirement already satisfied: mypy-extensions>=0.3.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from typing-inspect>=0.8.0->llama-index-core<0.11.0,>=0.10.1->llama-index-llms-bedrock) (1.0.0)\n",
|
||||
"Requirement already satisfied: marshmallow<4.0.0,>=3.18.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from dataclasses-json->llama-index-core<0.11.0,>=0.10.1->llama-index-llms-bedrock) (3.21.1)\n",
|
||||
"Requirement already satisfied: pytz>=2020.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from pandas->llama-index-core<0.11.0,>=0.10.1->llama-index-llms-bedrock) (2024.1)\n",
|
||||
"Requirement already satisfied: tzdata>=2022.7 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from pandas->llama-index-core<0.11.0,>=0.10.1->llama-index-llms-bedrock) (2024.1)\n",
|
||||
"Requirement already satisfied: packaging>=17.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from marshmallow<4.0.0,>=3.18.0->dataclasses-json->llama-index-core<0.11.0,>=0.10.1->llama-index-llms-bedrock) (24.0)\n",
|
||||
"Requirement already satisfied: annotated-types>=0.4.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from pydantic<3,>=1.9.0->anthropic<0.24.0,>=0.23.1->llama-index-llms-anthropic<0.2.0,>=0.1.7->llama-index-llms-bedrock) (0.6.0)\n",
|
||||
"Requirement already satisfied: pydantic-core==2.18.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from pydantic<3,>=1.9.0->anthropic<0.24.0,>=0.23.1->llama-index-llms-anthropic<0.2.0,>=0.1.7->llama-index-llms-bedrock) (2.18.1)\n",
|
||||
"Requirement already satisfied: six>=1.5 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.35.0,>=1.34.85->boto3<2.0.0,>=1.34.26->llama-index-llms-bedrock) (1.16.0)\n",
|
||||
"Requirement already satisfied: huggingface-hub<1.0,>=0.16.4 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from tokenizers>=0.13.0->anthropic<0.24.0,>=0.23.1->llama-index-llms-anthropic<0.2.0,>=0.1.7->llama-index-llms-bedrock) (0.22.2)\n",
|
||||
"Requirement already satisfied: filelock in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from huggingface-hub<1.0,>=0.16.4->tokenizers>=0.13.0->anthropic<0.24.0,>=0.23.1->llama-index-llms-anthropic<0.2.0,>=0.1.7->llama-index-llms-bedrock) (3.13.4)\n",
|
||||
"\n",
|
||||
"\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip available: \u001b[0m\u001b[31;49m22.3.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.0\u001b[0m\n",
|
||||
"\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n",
|
||||
"Note: you may need to restart the kernel to use updated packages.\n",
|
||||
"Requirement already satisfied: llama-index-embeddings-bedrock in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (0.1.4)\n",
|
||||
"Requirement already satisfied: boto3<2.0.0,>=1.34.23 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-embeddings-bedrock) (1.34.85)\n",
|
||||
"Requirement already satisfied: llama-index-core<0.11.0,>=0.10.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-embeddings-bedrock) (0.10.30)\n",
|
||||
"Requirement already satisfied: botocore<1.35.0,>=1.34.85 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from boto3<2.0.0,>=1.34.23->llama-index-embeddings-bedrock) (1.34.85)\n",
|
||||
"Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from boto3<2.0.0,>=1.34.23->llama-index-embeddings-bedrock) (1.0.1)\n",
|
||||
"Requirement already satisfied: s3transfer<0.11.0,>=0.10.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from boto3<2.0.0,>=1.34.23->llama-index-embeddings-bedrock) (0.10.1)\n",
|
||||
"Requirement already satisfied: PyYAML>=6.0.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-embeddings-bedrock) (6.0.1)\n",
|
||||
"Requirement already satisfied: SQLAlchemy[asyncio]>=1.4.49 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-embeddings-bedrock) (2.0.29)\n",
|
||||
"Requirement already satisfied: aiohttp<4.0.0,>=3.8.6 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-embeddings-bedrock) (3.9.4)\n",
|
||||
"Requirement already satisfied: dataclasses-json in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-embeddings-bedrock) (0.6.4)\n",
|
||||
"Requirement already satisfied: deprecated>=1.2.9.3 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-embeddings-bedrock) (1.2.14)\n",
|
||||
"Requirement already satisfied: dirtyjson<2.0.0,>=1.0.8 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-embeddings-bedrock) (1.0.8)\n",
|
||||
"Requirement already satisfied: fsspec>=2023.5.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-embeddings-bedrock) (2024.3.1)\n",
|
||||
"Requirement already satisfied: httpx in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-embeddings-bedrock) (0.27.0)\n",
|
||||
"Requirement already satisfied: llamaindex-py-client<0.2.0,>=0.1.18 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-embeddings-bedrock) (0.1.18)\n",
|
||||
"Requirement already satisfied: nest-asyncio<2.0.0,>=1.5.8 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-embeddings-bedrock) (1.6.0)\n",
|
||||
"Requirement already satisfied: networkx>=3.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-embeddings-bedrock) (3.3)\n",
|
||||
"Requirement already satisfied: nltk<4.0.0,>=3.8.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-embeddings-bedrock) (3.8.1)\n",
|
||||
"Requirement already satisfied: numpy in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-embeddings-bedrock) (1.26.4)\n",
|
||||
"Requirement already satisfied: openai>=1.1.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-embeddings-bedrock) (1.17.0)\n",
|
||||
"Requirement already satisfied: pandas in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-embeddings-bedrock) (2.2.2)\n",
|
||||
"Requirement already satisfied: pillow>=9.0.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-embeddings-bedrock) (10.3.0)\n",
|
||||
"Requirement already satisfied: requests>=2.31.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-embeddings-bedrock) (2.31.0)\n",
|
||||
"Requirement already satisfied: tenacity<9.0.0,>=8.2.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-embeddings-bedrock) (8.2.3)\n",
|
||||
"Requirement already satisfied: tiktoken>=0.3.3 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-embeddings-bedrock) (0.6.0)\n",
|
||||
"Requirement already satisfied: tqdm<5.0.0,>=4.66.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-embeddings-bedrock) (4.66.2)\n",
|
||||
"Requirement already satisfied: typing-extensions>=4.5.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-embeddings-bedrock) (4.11.0)\n",
|
||||
"Requirement already satisfied: typing-inspect>=0.8.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-embeddings-bedrock) (0.9.0)\n",
|
||||
"Requirement already satisfied: wrapt in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.1->llama-index-embeddings-bedrock) (1.16.0)\n",
|
||||
"Requirement already satisfied: aiosignal>=1.1.2 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core<0.11.0,>=0.10.1->llama-index-embeddings-bedrock) (1.3.1)\n",
|
||||
"Requirement already satisfied: attrs>=17.3.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core<0.11.0,>=0.10.1->llama-index-embeddings-bedrock) (23.2.0)\n",
|
||||
"Requirement already satisfied: frozenlist>=1.1.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core<0.11.0,>=0.10.1->llama-index-embeddings-bedrock) (1.4.1)\n",
|
||||
"Requirement already satisfied: multidict<7.0,>=4.5 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core<0.11.0,>=0.10.1->llama-index-embeddings-bedrock) (6.0.5)\n",
|
||||
"Requirement already satisfied: yarl<2.0,>=1.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core<0.11.0,>=0.10.1->llama-index-embeddings-bedrock) (1.9.4)\n",
|
||||
"Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from botocore<1.35.0,>=1.34.85->boto3<2.0.0,>=1.34.23->llama-index-embeddings-bedrock) (2.9.0.post0)\n",
|
||||
"Requirement already satisfied: urllib3!=2.2.0,<3,>=1.25.4 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from botocore<1.35.0,>=1.34.85->boto3<2.0.0,>=1.34.23->llama-index-embeddings-bedrock) (1.26.18)\n",
|
||||
"Requirement already satisfied: pydantic>=1.10 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llamaindex-py-client<0.2.0,>=0.1.18->llama-index-core<0.11.0,>=0.10.1->llama-index-embeddings-bedrock) (2.7.0)\n",
|
||||
"Requirement already satisfied: anyio in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from httpx->llama-index-core<0.11.0,>=0.10.1->llama-index-embeddings-bedrock) (4.3.0)\n",
|
||||
"Requirement already satisfied: certifi in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from httpx->llama-index-core<0.11.0,>=0.10.1->llama-index-embeddings-bedrock) (2024.2.2)\n",
|
||||
"Requirement already satisfied: httpcore==1.* in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from httpx->llama-index-core<0.11.0,>=0.10.1->llama-index-embeddings-bedrock) (1.0.5)\n",
|
||||
"Requirement already satisfied: idna in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from httpx->llama-index-core<0.11.0,>=0.10.1->llama-index-embeddings-bedrock) (3.7)\n",
|
||||
"Requirement already satisfied: sniffio in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from httpx->llama-index-core<0.11.0,>=0.10.1->llama-index-embeddings-bedrock) (1.3.1)\n",
|
||||
"Requirement already satisfied: h11<0.15,>=0.13 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from httpcore==1.*->httpx->llama-index-core<0.11.0,>=0.10.1->llama-index-embeddings-bedrock) (0.14.0)\n",
|
||||
"Requirement already satisfied: click in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from nltk<4.0.0,>=3.8.1->llama-index-core<0.11.0,>=0.10.1->llama-index-embeddings-bedrock) (8.1.7)\n",
|
||||
"Requirement already satisfied: joblib in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from nltk<4.0.0,>=3.8.1->llama-index-core<0.11.0,>=0.10.1->llama-index-embeddings-bedrock) (1.4.0)\n",
|
||||
"Requirement already satisfied: regex>=2021.8.3 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from nltk<4.0.0,>=3.8.1->llama-index-core<0.11.0,>=0.10.1->llama-index-embeddings-bedrock) (2023.12.25)\n",
|
||||
"Requirement already satisfied: distro<2,>=1.7.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from openai>=1.1.0->llama-index-core<0.11.0,>=0.10.1->llama-index-embeddings-bedrock) (1.9.0)\n",
|
||||
"Requirement already satisfied: charset-normalizer<4,>=2 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from requests>=2.31.0->llama-index-core<0.11.0,>=0.10.1->llama-index-embeddings-bedrock) (3.3.2)\n",
|
||||
"Requirement already satisfied: greenlet!=0.4.17 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from SQLAlchemy[asyncio]>=1.4.49->llama-index-core<0.11.0,>=0.10.1->llama-index-embeddings-bedrock) (3.0.3)\n",
|
||||
"Requirement already satisfied: mypy-extensions>=0.3.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from typing-inspect>=0.8.0->llama-index-core<0.11.0,>=0.10.1->llama-index-embeddings-bedrock) (1.0.0)\n",
|
||||
"Requirement already satisfied: marshmallow<4.0.0,>=3.18.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from dataclasses-json->llama-index-core<0.11.0,>=0.10.1->llama-index-embeddings-bedrock) (3.21.1)\n",
|
||||
"Requirement already satisfied: pytz>=2020.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from pandas->llama-index-core<0.11.0,>=0.10.1->llama-index-embeddings-bedrock) (2024.1)\n",
|
||||
"Requirement already satisfied: tzdata>=2022.7 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from pandas->llama-index-core<0.11.0,>=0.10.1->llama-index-embeddings-bedrock) (2024.1)\n",
|
||||
"Requirement already satisfied: packaging>=17.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from marshmallow<4.0.0,>=3.18.0->dataclasses-json->llama-index-core<0.11.0,>=0.10.1->llama-index-embeddings-bedrock) (24.0)\n",
|
||||
"Requirement already satisfied: annotated-types>=0.4.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from pydantic>=1.10->llamaindex-py-client<0.2.0,>=0.1.18->llama-index-core<0.11.0,>=0.10.1->llama-index-embeddings-bedrock) (0.6.0)\n",
|
||||
"Requirement already satisfied: pydantic-core==2.18.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from pydantic>=1.10->llamaindex-py-client<0.2.0,>=0.1.18->llama-index-core<0.11.0,>=0.10.1->llama-index-embeddings-bedrock) (2.18.1)\n",
|
||||
"Requirement already satisfied: six>=1.5 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.35.0,>=1.34.85->boto3<2.0.0,>=1.34.23->llama-index-embeddings-bedrock) (1.16.0)\n",
|
||||
"\n",
|
||||
"\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip available: \u001b[0m\u001b[31;49m22.3.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.0\u001b[0m\n",
|
||||
"\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n",
|
||||
"Note: you may need to restart the kernel to use updated packages.\n",
|
||||
"Requirement already satisfied: llama-index-retrievers-bedrock in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (0.1.1)\n",
|
||||
"Requirement already satisfied: llama-index-core<0.11.0,>=0.10.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-retrievers-bedrock) (0.10.30)\n",
|
||||
"Requirement already satisfied: PyYAML>=6.0.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (6.0.1)\n",
|
||||
"Requirement already satisfied: SQLAlchemy[asyncio]>=1.4.49 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (2.0.29)\n",
|
||||
"Requirement already satisfied: aiohttp<4.0.0,>=3.8.6 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (3.9.4)\n",
|
||||
"Requirement already satisfied: dataclasses-json in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (0.6.4)\n",
|
||||
"Requirement already satisfied: deprecated>=1.2.9.3 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (1.2.14)\n",
|
||||
"Requirement already satisfied: dirtyjson<2.0.0,>=1.0.8 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (1.0.8)\n",
|
||||
"Requirement already satisfied: fsspec>=2023.5.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (2024.3.1)\n",
|
||||
"Requirement already satisfied: httpx in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (0.27.0)\n",
|
||||
"Requirement already satisfied: llamaindex-py-client<0.2.0,>=0.1.18 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (0.1.18)\n",
|
||||
"Requirement already satisfied: nest-asyncio<2.0.0,>=1.5.8 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (1.6.0)\n",
|
||||
"Requirement already satisfied: networkx>=3.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (3.3)\n",
|
||||
"Requirement already satisfied: nltk<4.0.0,>=3.8.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (3.8.1)\n",
|
||||
"Requirement already satisfied: numpy in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (1.26.4)\n",
|
||||
"Requirement already satisfied: openai>=1.1.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (1.17.0)\n",
|
||||
"Requirement already satisfied: pandas in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (2.2.2)\n",
|
||||
"Requirement already satisfied: pillow>=9.0.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (10.3.0)\n",
|
||||
"Requirement already satisfied: requests>=2.31.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (2.31.0)\n",
|
||||
"Requirement already satisfied: tenacity<9.0.0,>=8.2.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (8.2.3)\n",
|
||||
"Requirement already satisfied: tiktoken>=0.3.3 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (0.6.0)\n",
|
||||
"Requirement already satisfied: tqdm<5.0.0,>=4.66.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (4.66.2)\n",
|
||||
"Requirement already satisfied: typing-extensions>=4.5.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (4.11.0)\n",
|
||||
"Requirement already satisfied: typing-inspect>=0.8.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (0.9.0)\n",
|
||||
"Requirement already satisfied: wrapt in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (1.16.0)\n",
|
||||
"Requirement already satisfied: aiosignal>=1.1.2 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (1.3.1)\n",
|
||||
"Requirement already satisfied: attrs>=17.3.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (23.2.0)\n",
|
||||
"Requirement already satisfied: frozenlist>=1.1.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (1.4.1)\n",
|
||||
"Requirement already satisfied: multidict<7.0,>=4.5 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (6.0.5)\n",
|
||||
"Requirement already satisfied: yarl<2.0,>=1.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (1.9.4)\n",
|
||||
"Requirement already satisfied: pydantic>=1.10 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llamaindex-py-client<0.2.0,>=0.1.18->llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (2.7.0)\n",
|
||||
"Requirement already satisfied: anyio in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from httpx->llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (4.3.0)\n",
|
||||
"Requirement already satisfied: certifi in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from httpx->llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (2024.2.2)\n",
|
||||
"Requirement already satisfied: httpcore==1.* in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from httpx->llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (1.0.5)\n",
|
||||
"Requirement already satisfied: idna in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from httpx->llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (3.7)\n",
|
||||
"Requirement already satisfied: sniffio in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from httpx->llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (1.3.1)\n",
|
||||
"Requirement already satisfied: h11<0.15,>=0.13 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from httpcore==1.*->httpx->llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (0.14.0)\n",
|
||||
"Requirement already satisfied: click in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from nltk<4.0.0,>=3.8.1->llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (8.1.7)\n",
|
||||
"Requirement already satisfied: joblib in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from nltk<4.0.0,>=3.8.1->llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (1.4.0)\n",
|
||||
"Requirement already satisfied: regex>=2021.8.3 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from nltk<4.0.0,>=3.8.1->llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (2023.12.25)\n",
|
||||
"Requirement already satisfied: distro<2,>=1.7.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from openai>=1.1.0->llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (1.9.0)\n",
|
||||
"Requirement already satisfied: charset-normalizer<4,>=2 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from requests>=2.31.0->llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (3.3.2)\n",
|
||||
"Requirement already satisfied: urllib3<3,>=1.21.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from requests>=2.31.0->llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (1.26.18)\n",
|
||||
"Requirement already satisfied: greenlet!=0.4.17 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from SQLAlchemy[asyncio]>=1.4.49->llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (3.0.3)\n",
|
||||
"Requirement already satisfied: mypy-extensions>=0.3.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from typing-inspect>=0.8.0->llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (1.0.0)\n",
|
||||
"Requirement already satisfied: marshmallow<4.0.0,>=3.18.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from dataclasses-json->llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (3.21.1)\n",
|
||||
"Requirement already satisfied: python-dateutil>=2.8.2 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from pandas->llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (2.9.0.post0)\n",
|
||||
"Requirement already satisfied: pytz>=2020.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from pandas->llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (2024.1)\n",
|
||||
"Requirement already satisfied: tzdata>=2022.7 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from pandas->llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (2024.1)\n",
|
||||
"Requirement already satisfied: packaging>=17.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from marshmallow<4.0.0,>=3.18.0->dataclasses-json->llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (24.0)\n",
|
||||
"Requirement already satisfied: annotated-types>=0.4.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from pydantic>=1.10->llamaindex-py-client<0.2.0,>=0.1.18->llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (0.6.0)\n",
|
||||
"Requirement already satisfied: pydantic-core==2.18.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from pydantic>=1.10->llamaindex-py-client<0.2.0,>=0.1.18->llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (2.18.1)\n",
|
||||
"Requirement already satisfied: six>=1.5 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from python-dateutil>=2.8.2->pandas->llama-index-core<0.11.0,>=0.10.0->llama-index-retrievers-bedrock) (1.16.0)\n",
|
||||
"\n",
|
||||
"\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip available: \u001b[0m\u001b[31;49m22.3.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.0\u001b[0m\n",
|
||||
"\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n",
|
||||
"Note: you may need to restart the kernel to use updated packages.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"%pip install llama-index\n",
|
||||
"%pip install llama-index-llms-bedrock\n",
|
||||
"%pip install llama-index-embeddings-bedrock\n",
|
||||
"%pip install llama-index-retrievers-bedrock"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "55c5a903-fdcb-4326-a0d4-8e44e2895e91",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Setup and imports"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 35,
|
||||
"id": "c19fc0a8-06fc-435c-9298-5632d81902e9",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from llama_index.core import Settings\n",
|
||||
"from llama_index.core.query_engine import RetrieverQueryEngine\n",
|
||||
"\n",
|
||||
"from llama_index.retrievers.bedrock import AmazonKnowledgeBasesRetriever\n",
|
||||
"from llama_index.llms.bedrock import Bedrock\n",
|
||||
"from llama_index.embeddings.bedrock import BedrockEmbedding"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 37,
|
||||
"id": "487a6e1d-f211-46bb-acb1-850eb31602b3",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"llm = Bedrock(model = \"anthropic.claude-v2\")\n",
|
||||
"embed_model = BedrockEmbedding(model = \"amazon.titan-embed-text-v1\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 39,
|
||||
"id": "2ab469e8-245d-4cf5-a385-23f2046b99fb",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"Settings.llm = llm\n",
|
||||
"Settings.embed_model = embed_model\n",
|
||||
"Settings.chunk_size = 512"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "2cee8d0c-dfe2-417f-8b24-079379822651",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Download data"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 40,
|
||||
"id": "97e0a9c7-7217-4475-86eb-468ebdc16658",
|
||||
"metadata": {
|
||||
"collapsed": true,
|
||||
"jupyter": {
|
||||
"outputs_hidden": true
|
||||
},
|
||||
"scrolled": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"--2024-04-18 13:29:33-- https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/10k/lyft_2021.pdf\n",
|
||||
"Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 2606:50c0:8001::154, 2606:50c0:8002::154, 2606:50c0:8003::154, ...\n",
|
||||
"Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|2606:50c0:8001::154|:443... connected.\n",
|
||||
"HTTP request sent, awaiting response... 200 OK\n",
|
||||
"Length: 1440303 (1.4M) [application/octet-stream]\n",
|
||||
"Saving to: ‘./data/10k/lyft_2021.pdf’\n",
|
||||
"\n",
|
||||
"./data/10k/lyft_202 100%[===================>] 1.37M --.-KB/s in 0.1s \n",
|
||||
"\n",
|
||||
"2024-04-18 13:29:33 (13.8 MB/s) - ‘./data/10k/lyft_2021.pdf’ saved [1440303/1440303]\n",
|
||||
"\n",
|
||||
"--2024-04-18 13:29:33-- https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/10k/uber_2021.pdf\n",
|
||||
"Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 2606:50c0:8001::154, 2606:50c0:8002::154, 2606:50c0:8003::154, ...\n",
|
||||
"Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|2606:50c0:8001::154|:443... connected.\n",
|
||||
"HTTP request sent, awaiting response... 200 OK\n",
|
||||
"Length: 1880483 (1.8M) [application/octet-stream]\n",
|
||||
"Saving to: ‘./data/10k/uber_2021.pdf’\n",
|
||||
"\n",
|
||||
"./data/10k/uber_202 100%[===================>] 1.79M --.-KB/s in 0.08s \n",
|
||||
"\n",
|
||||
"2024-04-18 13:29:34 (22.0 MB/s) - ‘./data/10k/uber_2021.pdf’ saved [1880483/1880483]\n",
|
||||
"\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"!mkdir -p './data/10k/'\n",
|
||||
"!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/10k/lyft_2021.pdf' -O './data/10k/lyft_2021.pdf'\n",
|
||||
"!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/10k/uber_2021.pdf' -O './data/10k/uber_2021.pdf'"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"id": "d45ec59f-5e59-40ca-8016-e2d3796ff9fa",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"knowledge_base_id = \"PO6XCEKGI1\"\n",
|
||||
"top_k = 4\n",
|
||||
"search_mode = \"HYBRID\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"id": "09dbcf64-181a-46a7-ab10-3c2b250c859f",
|
||||
"metadata": {
|
||||
"jupyter": {
|
||||
"source_hidden": true
|
||||
}
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"apple_fpath = \"data/apple_2019.pdf\"\n",
|
||||
"apple_retriever = AmazonKnowledgeBasesRetriever(\n",
|
||||
" knowledge_base_id=knowledge_base_id,\n",
|
||||
" retrieval_config={\n",
|
||||
" \"vectorSearchConfiguration\": {\n",
|
||||
" \"numberOfResults\": top_k,\n",
|
||||
" \"overrideSearchType\": search_mode,\n",
|
||||
" \"filter\": {\"equals\": {\"key\": \"file_path\", \"value\": apple_fpath}},\n",
|
||||
" }\n",
|
||||
" },\n",
|
||||
")\n",
|
||||
"apple_engine = RetrieverQueryEngine(retriever=apple_retriever)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"id": "8e5c7d0a-4760-4f06-9f36-10193c55ecfe",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"tesla_fpath = \"data/tesla_2019.pdf\"\n",
|
||||
"tesla_retriever = AmazonKnowledgeBasesRetriever(\n",
|
||||
" knowledge_base_id=knowledge_base_id,\n",
|
||||
" retrieval_config={\n",
|
||||
" \"vectorSearchConfiguration\": {\n",
|
||||
" \"numberOfResults\": top_k,\n",
|
||||
" \"overrideSearchType\": search_mode,\n",
|
||||
" \"filter\": {\"equals\": {\"key\": \"file_path\", \"value\": tesla_fpath}},\n",
|
||||
" }\n",
|
||||
" },\n",
|
||||
")\n",
|
||||
"tesla_engine = RetrieverQueryEngine(retriever=tesla_retriever)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"id": "91c0f3bb-ceea-41c0-82ce-ea337df24dfc",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from llama_index.core.tools import QueryEngineTool, ToolMetadata"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 24,
|
||||
"id": "3872f957-68ce-45dc-a152-62e64ac31702",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"tools = [\n",
|
||||
" QueryEngineTool(\n",
|
||||
" query_engine=apple_engine,\n",
|
||||
" metadata=ToolMetadata(\n",
|
||||
" name=\"apple_2019\",\n",
|
||||
" description=\"10K filing for Apple 2019\",\n",
|
||||
" ),\n",
|
||||
" ),\n",
|
||||
" QueryEngineTool(\n",
|
||||
" query_engine=tesla_engine,\n",
|
||||
" metadata=ToolMetadata(\n",
|
||||
" name=\"tesla_2019\",\n",
|
||||
" description=\"10K filing for Tesla 2019\",\n",
|
||||
" ),\n",
|
||||
" ),\n",
|
||||
"]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 25,
|
||||
"id": "8c388cbc-24aa-40ad-9a38-3e4497b6df93",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from llama_index.core.query_engine import SubQuestionQueryEngine"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 26,
|
||||
"id": "9a961416-717e-4bc1-bf19-c78e9b4ee554",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"query_engine = SubQuestionQueryEngine.from_defaults(\n",
|
||||
" query_engine_tools=tools,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 27,
|
||||
"id": "8e1947b9-b218-4a9f-ab46-91b59bc7e214",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Generated 2 sub questions.\n",
|
||||
"\u001b[1;3;38;2;237;90;200m[apple_2019] Q: What were the key risk areas identified in the 10K filing for Apple in 2019?\n",
|
||||
"\u001b[0m\u001b[1;3;38;2;90;149;237m[tesla_2019] Q: What were the key risk areas identified in the 10K filing for Tesla in 2019?\n",
|
||||
"\u001b[0m\u001b[1;3;38;2;237;90;200m[apple_2019] A: Key risk areas identified in the 10K filing for Apple in 2019 included competitive markets, credit risk associated with derivative instruments, potential legal and other claims including intellectual property rights infringement, and exposure to complex and changing laws and regulations worldwide.\n",
|
||||
"\u001b[0m\u001b[1;3;38;2;90;149;237m[tesla_2019] A: The key risk areas identified in the 10K filing for Tesla in 2019 included automotive revenue recognition controls for sales with resale value guarantees or buyback options, sales return reserves, management's estimation of future market values, historical experience evaluation, economic incentives for customers, warranty reserves for new and used vehicles, and interest rate risk related to borrowings with floating rates.\n",
|
||||
"\u001b[0m"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"response = await query_engine.aquery('compare risks areas for apple and tesla in 2019')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 30,
|
||||
"id": "e320d865-8101-41aa-88c2-c1752193dc7e",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from llama_index.core.response.pprint_utils import pprint_response"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 29,
|
||||
"id": "6f37234d-9322-48f4-b03f-d034757716ab",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Final Response: The key risk areas identified in the 10K filing for\n",
|
||||
"Apple in 2019 included competitive markets, credit risk associated\n",
|
||||
"with derivative instruments, potential legal and other claims\n",
|
||||
"including intellectual property rights infringement, and exposure to\n",
|
||||
"complex and changing laws and regulations worldwide. On the other\n",
|
||||
"hand, the key risk areas identified in the 10K filing for Tesla in\n",
|
||||
"2019 included automotive revenue recognition controls for sales with\n",
|
||||
"resale value guarantees or buyback options, sales return reserves,\n",
|
||||
"management's estimation of future market values, historical experience\n",
|
||||
"evaluation, economic incentives for customers, warranty reserves for\n",
|
||||
"new and used vehicles, and interest rate risk related to borrowings\n",
|
||||
"with floating rates.\n",
|
||||
"______________________________________________________________________\n",
|
||||
"Source Node 1/10\n",
|
||||
"Node ID: aef1d97a-4824-415a-81e1-a55f62041c40\n",
|
||||
"Similarity: None\n",
|
||||
"Text: Sub question: What were the key risk areas identified in the 10K\n",
|
||||
"filing for Apple in 2019? Response: Key risk areas identified in the\n",
|
||||
"10K filing for Apple in 2019 included competitive markets, credit risk\n",
|
||||
"associated with derivative instruments, potential legal and other\n",
|
||||
"claims including intellectual property rights infringement, and\n",
|
||||
"exposure to ...\n",
|
||||
"______________________________________________________________________\n",
|
||||
"Source Node 2/10\n",
|
||||
"Node ID: 400270b3-f089-4b86-b736-7bdd311d97f5\n",
|
||||
"Similarity: None\n",
|
||||
"Text: Sub question: What were the key risk areas identified in the 10K\n",
|
||||
"filing for Tesla in 2019? Response: The key risk areas identified in\n",
|
||||
"the 10K filing for Tesla in 2019 included automotive revenue\n",
|
||||
"recognition controls for sales with resale value guarantees or buyback\n",
|
||||
"options, sales return reserves, management's estimation of future\n",
|
||||
"market values, ...\n",
|
||||
"______________________________________________________________________\n",
|
||||
"Source Node 3/10\n",
|
||||
"Node ID: cefcebde-22f0-4309-b735-ea63829d5c54\n",
|
||||
"Similarity: 0.56899893\n",
|
||||
"Text: Licensing The Company licenses the use of certain of its\n",
|
||||
"intellectual property, and provides other related services. Other\n",
|
||||
"Services The Company delivers a variety of other services available\n",
|
||||
"in certain countries, including Apple Arcade™, a game subscription\n",
|
||||
"service; Apple Card™, a co-branded credit card; Apple News+, a\n",
|
||||
"subscription news and m...\n",
|
||||
"______________________________________________________________________\n",
|
||||
"Source Node 4/10\n",
|
||||
"Node ID: 12ceba11-656d-473f-b1a1-8fdc76e0bef8\n",
|
||||
"Similarity: 0.56899893\n",
|
||||
"Text: The credit risk amounts represent the Company’s gross exposure\n",
|
||||
"to potential accounting loss on derivative instruments that are\n",
|
||||
"outstanding or unsettled if all counterparties failed to perform\n",
|
||||
"according to the terms of the contract, based on then-current currency\n",
|
||||
"or interest rates at each respective date. The Company’s exposure to\n",
|
||||
"credit loss and...\n",
|
||||
"______________________________________________________________________\n",
|
||||
"Source Node 5/10\n",
|
||||
"Node ID: 1f30a6e5-2fac-444c-a562-ed1432348c2c\n",
|
||||
"Similarity: 0.5647855781355375\n",
|
||||
"Text: The SEC maintains an Internet site that contains reports, proxy\n",
|
||||
"and information statements, and other information regarding issuers\n",
|
||||
"that file electronically with the SEC at www.sec.gov. The Company\n",
|
||||
"periodically provides other information for investors on its corporate\n",
|
||||
"website, www.apple.com, and its investor relations website,\n",
|
||||
"investor.apple.com...\n",
|
||||
"______________________________________________________________________\n",
|
||||
"Source Node 6/10\n",
|
||||
"Node ID: e2fb85bc-b73e-4cf5-be8e-28c62e4c8f50\n",
|
||||
"Similarity: 0.55775213\n",
|
||||
"Text: Except as described in Part II, Item 8 of this Form 10-K in the\n",
|
||||
"Notes to Consolidated Financial Statements in Note 10, “Commitments\n",
|
||||
"and Contingencies” under the heading “Contingencies,” in the opinion\n",
|
||||
"of management, there was not at least a reasonable possibility the\n",
|
||||
"Company may have incurred a material loss, or a material loss greater\n",
|
||||
"than a re...\n",
|
||||
"______________________________________________________________________\n",
|
||||
"Source Node 7/10\n",
|
||||
"Node ID: b34fa82b-56af-4ad1-b290-e0599be74894\n",
|
||||
"Similarity: 0.565212\n",
|
||||
"Text: These procedures included testing the effectiveness of controls\n",
|
||||
"relating to automotive revenue recognition for sales to customers with\n",
|
||||
"a resale value guarantee or buyback option as well as the related\n",
|
||||
"sales return reserve, including controls over management’s estimate of\n",
|
||||
"expected future market values and historical experience. These\n",
|
||||
"procedures a...\n",
|
||||
"______________________________________________________________________\n",
|
||||
"Source Node 8/10\n",
|
||||
"Node ID: 187534f6-30ad-4867-9834-856c507e6cd4\n",
|
||||
"Similarity: 0.565212\n",
|
||||
"Text: # SECURITIES AND EXCHANGE COMMISSIONUNITED STATES Washington,\n",
|
||||
"D.C. 20549 ## FORM 10-K (Mark One) ☒ ANNUAL REPORT PURSUANT TO\n",
|
||||
"SECTION 13 OR 15(d) OF THE SECURITIES EXCHANGE ACT OF 1934 For the\n",
|
||||
"fiscal year ended OR December 31, 2019 ☐ TRANSITION REPORT PURSUANT\n",
|
||||
"TO SECTION 13 OR 15(d) OF THE SECURITIES EXCHANGE ACT OF 1934 For the\n",
|
||||
"transition...\n",
|
||||
"______________________________________________________________________\n",
|
||||
"Source Node 9/10\n",
|
||||
"Node ID: aa50de0c-2047-4bd3-9752-9a1aa1589488\n",
|
||||
"Similarity: 0.5628578\n",
|
||||
"Text: /s/ PricewaterhouseCoopers LLP San Jose, California February\n",
|
||||
"13, 2020 CERTIFICATIONS Exhibit 31.1 I, Elon Musk, certify that: I\n",
|
||||
"have reviewed this Annual Report on Form 10-K of Tesla, Inc.; Based on\n",
|
||||
"my knowledge, this report does not contain any untrue statement of a\n",
|
||||
"material fact or omit to state a material fact necessary to make the\n",
|
||||
"statem...\n",
|
||||
"______________________________________________________________________\n",
|
||||
"Source Node 10/10\n",
|
||||
"Node ID: e2ad29a3-ca3a-4ef8-81df-1dee85a72cf9\n",
|
||||
"Similarity: 0.5622827\n",
|
||||
"Text: These changes would have resulted in an adverse impact of $362\n",
|
||||
"million at December 31, 2019 and $176 million at December 31, 2018\n",
|
||||
"assuming no foreign currency hedging. Interest Rate Risk We are\n",
|
||||
"exposed to interest rate risk on our borrowings that bear interest at\n",
|
||||
"floating rates. Pursuant to our risk management policies, in certain\n",
|
||||
"cases, we ut...\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"pprint_response(response, show_source=True)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "941412b8-ee09-4ac7-ae1d-d99c387d7a76",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.11.3"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@@ -1,3 +0,0 @@
|
||||
# rag-bedrock
|
||||
|
||||
Example notebooks for building RAG applications with LlamaIndex and Bedrock (LLM, embeddinng model, KB)
|
||||
Reference in New Issue
Block a user