mirror of
https://github.com/run-llama/llama_cloud_services.git
synced 2026-07-22 12:35:50 -04:00
Compare commits
1 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Logan Markewich
|
fa545c0d50 |
@@ -7,6 +7,8 @@ assignees: ''
|
||||
|
||||
---
|
||||
|
||||
_Note: we're aware of some missing content in the output and layout issues on tables. Please refrain from opening new issues on this topic unless if you think it's different from what has already been reported._
|
||||
|
||||
**Describe the bug**
|
||||
Write a concise description of what the bug is.
|
||||
|
||||
@@ -17,15 +19,19 @@ If possible, please provide the PDF file causing the issue.
|
||||
If you have it, please provide the ID of the job you ran.
|
||||
You can find it here: https://cloud.llamaindex.ai/parse in the "History" tab.
|
||||
|
||||
**Screenshots**
|
||||
Feel free to also provide screenshots if relevant.
|
||||
|
||||
**Client:**
|
||||
Please remove untested options:
|
||||
- Python Library
|
||||
- API
|
||||
- Frontend (cloud.llamaindex.ai)
|
||||
- Python Library
|
||||
- Typescript Library
|
||||
- Notebook
|
||||
- API
|
||||
|
||||
**Options**
|
||||
What options did you use? Multimodal, fast mode, parsing instructions, etc.
|
||||
|
||||
**Additional context**
|
||||
Add any additional context about the problem here.
|
||||
What options did you use? Premium mode, multimodal, fast mode, parsing instructions, etc.
|
||||
Screenshots, code snippets, etc.
|
||||
|
||||
@@ -17,7 +17,7 @@ jobs:
|
||||
# You can use PyPy versions in python-version.
|
||||
# For example, pypy-2.7 and pypy-3.8
|
||||
matrix:
|
||||
python-version: ["3.9", "3.10", "3.11", "3.12"]
|
||||
python-version: ["3.8", "3.10", "3.11"]
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
with:
|
||||
|
||||
@@ -38,22 +38,7 @@ Lastly, install the package:
|
||||
|
||||
`pip install llama-parse`
|
||||
|
||||
Now you can parse your first PDF file using the command line interface. Use the command `llama-parse [file_paths]`. See the help text with `llama-parse --help`.
|
||||
|
||||
```bash
|
||||
export LLAMA_CLOUD_API_KEY='llx-...'
|
||||
|
||||
# output as text
|
||||
llama-parse my_file.pdf --result-type text --output-file output.txt
|
||||
|
||||
# output as markdown
|
||||
llama-parse my_file.pdf --result-type markdown --output-file output.md
|
||||
|
||||
# output as raw json
|
||||
llama-parse my_file.pdf --output-raw-json --output-file output.json
|
||||
```
|
||||
|
||||
You can also create simple scripts:
|
||||
Now you can run the following to parse your first PDF file:
|
||||
|
||||
```python
|
||||
import nest_asyncio
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
Binary file not shown.
|
Before Width: | Height: | Size: 6.9 MiB |
File diff suppressed because one or more lines are too long
@@ -342,7 +342,7 @@
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"display_name": "llama-parse-aNC435Vv-py3.10",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,357 +0,0 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "97c79c38-38a3-40f3-ba2e-250649347d63",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"<a href=\"https://colab.research.google.com/github/run-llama/llama_parse/blob/main/examples/demo_starter_multimodal.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "4e081457",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Multimodal Parsing using LlamaParse\n",
|
||||
"\n",
|
||||
"This cookbook shows you how to use LlamaParse to parse any document with the multimodal capabilities of Multi-Modal LLMs from Anthropic/ OpenAI.\n",
|
||||
"\n",
|
||||
"LlamaParse allows you to plug in external, multimodal model vendors for parsing - we handle the error correction, validation, and scalability/reliability for you.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "qOdqBxCS51Ow",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Installation"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "H_Vqcylb50vm",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!pip install llama-parse"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "15e60ecf-519c-41fc-911b-765adaf8bad4",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Setup\n",
|
||||
"\n",
|
||||
"Here we setup `LLAMA_CLOUD_API_KEY` for using `LlamaParse`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "91a9e532-1454-40e0-bbf0-fd442c350121",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import nest_asyncio\n",
|
||||
"\n",
|
||||
"nest_asyncio.apply()\n",
|
||||
"\n",
|
||||
"import os\n",
|
||||
"\n",
|
||||
"# API access to llama-cloud\n",
|
||||
"os.environ[\"LLAMA_CLOUD_API_KEY\"] = \"<YOUR LLAMACLOUD API KEY>\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "LGwBNPNotZRQ",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Download Data\n",
|
||||
"\n",
|
||||
"For this demonstration, we will use OpenAI's recent paper `Evaluation of OpenAI o1: Opportunities and Challenges of AGI`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "IjtKDQRLrylI",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"--2024-12-05 18:54:24-- https://arxiv.org/pdf/2409.18486\n",
|
||||
"Resolving arxiv.org (arxiv.org)... 151.101.67.42, 151.101.131.42, 151.101.3.42, ...\n",
|
||||
"Connecting to arxiv.org (arxiv.org)|151.101.67.42|:443... connected.\n",
|
||||
"HTTP request sent, awaiting response... 200 OK\n",
|
||||
"Length: 13986265 (13M) [application/pdf]\n",
|
||||
"Saving to: ‘o1.pdf’\n",
|
||||
"\n",
|
||||
"o1.pdf 100%[===================>] 13.34M 11.8MB/s in 1.1s \n",
|
||||
"\n",
|
||||
"2024-12-05 18:54:26 (11.8 MB/s) - ‘o1.pdf’ saved [13986265/13986265]\n",
|
||||
"\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"!wget \"https://arxiv.org/pdf/2409.18486\" -O \"o1.pdf\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "4e29a9d7-5bd9-4fb8-8ec1-4c128a748662",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Initialize LlamaParse\n",
|
||||
"\n",
|
||||
"Initialize LlamaParse in multimodal mode, and specify the vendor.\n",
|
||||
"\n",
|
||||
"**NOTE**: optionally you can specify the Anthropic/ OpenAI API key. If you choose to do so LlamaParse will only charge you 1 credit (0.3c) per page. \n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Using your own API key may incur additional costs from your model provider and could result in failed pages or documents if you do not have sufficient usage limits."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "dc921729-3446-42ca-8e1b-a6fd26195ed9",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from llama_index.core.schema import TextNode\n",
|
||||
"from typing import List\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def get_text_nodes(json_list: List[dict]):\n",
|
||||
" text_nodes = []\n",
|
||||
" for idx, page in enumerate(json_list):\n",
|
||||
" text_node = TextNode(text=page[\"md\"], metadata={\"page\": page[\"page\"]})\n",
|
||||
" text_nodes.append(text_node)\n",
|
||||
" return text_nodes"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "1b5d6da6",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### With anthropic-sonnet-3.5"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "f2e9d9cf-8189-4fcb-b34f-cde6cc0b59c8",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Started parsing the file under job_id dd9d5e0f-160e-486a-89a2-6005e5a1c2ac\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from llama_parse import LlamaParse\n",
|
||||
"\n",
|
||||
"parser = LlamaParse(\n",
|
||||
" result_type=\"markdown\",\n",
|
||||
" use_vendor_multimodal_model=True,\n",
|
||||
" vendor_multimodal_model_name=\"anthropic-sonnet-3.5\",\n",
|
||||
" target_pages=\"24\"\n",
|
||||
" # invalidate_cache=True\n",
|
||||
")\n",
|
||||
"json_objs = parser.get_json_result(\"o1.pdf\")\n",
|
||||
"json_list = json_objs[0][\"pages\"]\n",
|
||||
"docs = get_text_nodes(json_list)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "4f3c51b0-7878-48d7-9bc3-02b516500128",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### With GPT-4o\n",
|
||||
"\n",
|
||||
"For comparison, we will also parse the document using GPT-4o."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "6fc3f258-50ae-4988-b904-c105463a498f",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Started parsing the file under job_id 6a4dea44-4f90-406b-b290-9e98620b1232\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from llama_parse import LlamaParse\n",
|
||||
"\n",
|
||||
"parser_gpt4o = LlamaParse(\n",
|
||||
" result_type=\"markdown\",\n",
|
||||
" use_vendor_multimodal_model=True,\n",
|
||||
" vendor_multimodal_model=\"openai-gpt4o\",\n",
|
||||
" target_pages=\"24\",\n",
|
||||
" # invalidate_cache=True\n",
|
||||
")\n",
|
||||
"json_objs_gpt4o = parser_gpt4o.get_json_result(\"o1.pdf\")\n",
|
||||
"json_list_gpt4o = json_objs_gpt4o[0][\"pages\"]\n",
|
||||
"docs_gpt4o = get_text_nodes(json_list_gpt4o)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "44c20f7a-2901-4dd0-b635-a4b33c5664c1",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### View Results\n",
|
||||
"\n",
|
||||
"Let's visualize the results along with the original document page."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "778698aa-da7e-4081-b3b5-0372f228536f",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"page: 25\n",
|
||||
"\n",
|
||||
"| Participant_ID | clinical Description Reference |\n",
|
||||
"|-----------------|----------------------------------|\n",
|
||||
"| Attribute | Value | Basic Personal Information: Subject 098_S_0896 is a 72.0-year-old Female who has completed 15 years of education. The ethnicity is Not Hisp/Latino and race is White. Marital status is Married. Initially diagnosed as AD, as of the date 2007-10-24, the final diagnosis was Dementia. |\n",
|
||||
"| Age | 72.0 |\n",
|
||||
"| Sex | Female |\n",
|
||||
"| Education | 15 |\n",
|
||||
"| Race | White | Biomarker Measurements: The subject's genetic profile includes an ApoE4 status of 0.0... |\n",
|
||||
"| DX_bl | AD |\n",
|
||||
"| DX | Dementia |\n",
|
||||
"| ... | ... | Cognitive and Neurofunctional Assessments: The Mini-Mental State Examination score stands at 29.0. The Clinical Dementia Rating, sum of boxes, is 1.0. ADAS 11 and 13 scores are 4.67 and 4.67 respectively, with a score of 1.0 in delayed word recall... |\n",
|
||||
"| APOE4 | 1.0 |\n",
|
||||
"| TAU | 212.5 |\n",
|
||||
"| ... | ... |\n",
|
||||
"| MMSE | 29.0 | Volumetric Data: Under MRI conditions at a field strength of 1.5 Tesla MRI Tesla, using Cross Sectional FreeSurfer (FreeSurfer Version 4.3), the imaging data recorded includes ventricles volume at 54422.0, hippocampus volume at 6677.0, whole brain volume at 1147980.0, entorhinal cortex volume at 2782.0, fusiform gyrus volume at 19432.0, and middle temporal area volume at 24951.0. The intracranial volume measured is 1799580.0.... |\n",
|
||||
"| CDRSB | 0.0 |\n",
|
||||
"| ... | ... |\n",
|
||||
"| FLDSTRENG | 1.5 Tesla MRI |\n",
|
||||
"| Ventricles | 84599 |\n",
|
||||
"| Hippocampus | 5319 |\n",
|
||||
"| ... | ... |\n",
|
||||
"\n",
|
||||
"Figure 2: An example of a patient table and its corresponding clinical description.\n",
|
||||
"\n",
|
||||
"skills. Mathematics, as a highly structured and logic-driven discipline, provides an ideal testing ground for evaluating this reasoning ability. To investigate o1-preview's performance, we designed a series of tests covering various difficulty levels. We begin with high school-level math competition problems in this section, followed by college-level mathematics problems in the next section, allowing us to observe the model's logical reasoning across varying levels of complexity.\n",
|
||||
"\n",
|
||||
"In this section, we selected two primary areas of mathematics: algebra and counting and probability in this section. We chose these two topics because of their heavy reliance on problem-solving skills and their frequent use in assessing logical and abstract thinking [46]. The dataset used in testing is from the MATH dataset [46]. The problems in the dataset cover a wide range of subjects, including Prealgebra, Intermediate Algebra, Algebra, Geometry, Counting and Probability, Number Theory, and Precalculus. Each problem is categorized based on difficulty, ranked from level 1 to 5, according to the Art of Problem Solving (AoPS). The dataset mainly comprises problems from various high school math competitions, including the American Mathematics Competitions (AMC) 10 and 12, as well as the American Invitational Mathematics Examination (AIME), and other similar contests. Each problem comes with detailed reference solutions, allowing for a comprehensive comparison of o1-preview's solutions.\n",
|
||||
"\n",
|
||||
"In addition to evaluating the final answers produced by o1-preview, our analysis delves into the step-by-step reasoning process of the o1-preview's solutions. By comparing o1-preview's solutions with the dataset's solutions, we assess its ability to engage in logical reasoning, handle abstract problem-solving tasks, and apply structured approaches to reach correct answers. This deeper analysis offers insights into o1-preview's overall reasoning capabilities, using mathematics as a reliable indicator for logical and structured thought processes.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# using Sonnet-3.5\n",
|
||||
"print(docs[0].get_content(metadata_mode=\"all\"))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "1511a30f-3efc-4142-9668-7dc056a24d0c",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"page: 25\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"| Participant_ID | clinical Description Reference |\n",
|
||||
"|----------------|--------------------------------|\n",
|
||||
"| **Attribute** | **Value** |\n",
|
||||
"| Age | 72.0 |\n",
|
||||
"| Sex | Female |\n",
|
||||
"| Education | 15 |\n",
|
||||
"| Race | White |\n",
|
||||
"| DX_bl | AD |\n",
|
||||
"| DX | Dementia |\n",
|
||||
"| ... | ... |\n",
|
||||
"| APOE4 | 1.0 |\n",
|
||||
"| TAU | 212.5 |\n",
|
||||
"| ... | ... |\n",
|
||||
"| MMSE | 29.0 |\n",
|
||||
"| CDRSB | 0.0 |\n",
|
||||
"| ... | ... |\n",
|
||||
"| FLDSTRENG | 1.5 Tesla MRI |\n",
|
||||
"| Ventricles | 84599 |\n",
|
||||
"| Hippocampus | 5319 |\n",
|
||||
"| ... | ... |\n",
|
||||
"\n",
|
||||
"**Basic Personal Information:** Subject 098_S_0896 is a 72.0-year-old Female who has completed 15 years of education. The ethnicity is Not Hisp/Latino and race is White. Marital status is Married. Initially diagnosed as AD, as of the date 2007-10-24, the final diagnosis was Dementia.\n",
|
||||
"\n",
|
||||
"**Biomarker Measurements:** The subject's genetic profile includes an ApoE4 status of 0.0...\n",
|
||||
"\n",
|
||||
"**Cognitive and Neurofunctional Assessments:** The Mini-Mental State Examination score stands at 29.0. The Clinical Dementia Rating, sum of boxes, is 1.0. ADAS 11 and 13 scores are 4.67 and 4.67 respectively, with a score of 1.0 in delayed word recall...\n",
|
||||
"\n",
|
||||
"**Volumetric Data:** Under MRI conditions at a field strength of 1.5 Tesla MRI Tesla, using Cross-Sectional FreeSurfer (FreeSurfer Version 4.3), the imaging data recorded includes ventricles volume at 84422.0, hippocampus volume at 6677.0, whole brain volume at 1147980.0, entorhinal cortex volume at 27820.0, fusiform gyrus volume at 19432.0, and middle temporal area volume at 24951.0. The intracranial volume measured is 1799580.0...\n",
|
||||
"\n",
|
||||
"Figure 2: An example of a patient table and its corresponding clinical description.\n",
|
||||
"\n",
|
||||
"----\n",
|
||||
"\n",
|
||||
"Skills. Mathematics, as a highly structured and logic-driven discipline, provides an ideal testing ground for evaluating this reasoning ability. To investigate o1-preview’s performance, we designed a series of tests covering various difficulty levels. We begin with high school-level math competition problems in this section, followed by college-level mathematics problems in the next section, allowing us to observe the model’s logical reasoning across varying levels of complexity.\n",
|
||||
"\n",
|
||||
"In this section, we selected two primary areas of mathematics: algebra and counting and probability in this section. We chose these two topics because of their heavy reliance on problem-solving skills and their frequent use in assessing logical and abstract thinking [46]. The dataset used in testing is from the MATH dataset [46]. The problems in the dataset cover a wide range of subjects, including Prealgebra, Intermediate Algebra, Algebra, Geometry, Counting and Probability, Number Theory, and Precalculus. Each problem is categorized based on difficulty, ranked from level 1 to 5, according to the Art of Problem Solving (AoPS). The dataset mainly comprises problems from various high school math competitions, including the American Mathematics Competitions (AMC) 10 and 12, as well as the American Invitational Mathematics Examination (AIME), and other similar contests. Each problem comes with detailed reference solutions, allowing for a comprehensive comparison of o1-preview’s solutions.\n",
|
||||
"\n",
|
||||
"In addition to evaluating the final answers produced by o1-preview, our analysis delves into the step-by-step reasoning process of the o1-preview’s solutions. By comparing o1-preview’s solutions with the dataset’s solutions, we assess its ability to engage in logical reasoning, handle abstract problem-solving tasks, and apply structured approaches to reach correct answers. This deeper analysis offers insights into o1-preview’s overall reasoning capabilities, using mathematics as a reliable indicator for logical and structured thought processes.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# using GPT-4o\n",
|
||||
"print(docs_gpt4o[0].get_content(metadata_mode=\"all\"))"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"provenance": []
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "llamacloud",
|
||||
"language": "python",
|
||||
"name": "llamacloud"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@@ -1,170 +0,0 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"<a href=\"https://colab.research.google.com/github/run-llama/llama_parse/blob/main/examples/demo_starter_parse_selected_pages.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Parse Selected Pages \n",
|
||||
"\n",
|
||||
"In this notebook we will demonstrate how to parse selected pages in a document using LlamaParse."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Installation\n",
|
||||
"\n",
|
||||
"Here we install `llama-parse` used for parsing the document"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!pip install llama-parse"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Set API Key"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# llama-parse is async-first, running the async code in a notebook requires the use of nest_asyncio\n",
|
||||
"import nest_asyncio\n",
|
||||
"\n",
|
||||
"nest_asyncio.apply()\n",
|
||||
"\n",
|
||||
"import os\n",
|
||||
"\n",
|
||||
"# API access to llama-cloud\n",
|
||||
"os.environ[\"LLAMA_CLOUD_API_KEY\"] = \"<YOUR LLAMACLOUD API KEY>\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Download Data\n",
|
||||
"\n",
|
||||
"Here we download Uber 2021 10K SEC filings data for the demonstration."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"--2024-12-05 11:40:59-- https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/10k/uber_2021.pdf\n",
|
||||
"Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 2606:50c0:8000::154, 2606:50c0:8002::154, 2606:50c0:8003::154, ...\n",
|
||||
"Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|2606:50c0:8000::154|:443... connected.\n",
|
||||
"HTTP request sent, awaiting response... 200 OK\n",
|
||||
"Length: 1880483 (1.8M) [application/octet-stream]\n",
|
||||
"Saving to: ‘./uber_2021.pdf’\n",
|
||||
"\n",
|
||||
"./uber_2021.pdf 100%[===================>] 1.79M --.-KB/s in 0.1s \n",
|
||||
"\n",
|
||||
"2024-12-05 11:40:59 (14.2 MB/s) - ‘./uber_2021.pdf’ saved [1880483/1880483]\n",
|
||||
"\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/10k/uber_2021.pdf' -O './uber_2021.pdf'"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Parse the PDF file in selected pages\n",
|
||||
"\n",
|
||||
"Here we will parse the PDF file in selected pages and get the text in `markdown` format."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Started parsing the file under job_id ad1087c1-b085-4dc7-9aa8-d13cdd440f2b\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from llama_parse import LlamaParse\n",
|
||||
"\n",
|
||||
"parser = LlamaParse(target_pages=\"0,1,2\", result_type=\"markdown\")\n",
|
||||
"\n",
|
||||
"documents = parser.load_data(\"./uber_2021.pdf\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[Document(id_='d0b34f4a-27ef-48e2-a92a-386e5e265f4c', embedding=None, metadata={}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={}, metadata_template='{key}: {value}', metadata_separator='\\n', text='# UNITED STATES SECURITIES AND EXCHANGE COMMISSION\\n\\n# Washington, D.C. 20549\\n\\n# FORM 10-K\\n\\n(Mark One)\\n\\n☒ ANNUAL REPORT PURSUANT TO SECTION 13 OR 15(d) OF THE SECURITIES EXCHANGE ACT OF 1934\\n\\nFor the fiscal year ended December 31, 2021\\n\\nOR\\n\\n☐ TRANSITION REPORT PURSUANT TO SECTION 13 OR 15(d) OF THE SECURITIES EXCHANGE ACT OF 1934\\n\\nFor the transition period from _____ to _____\\n\\nCommission File Number: 001-38902\\n\\n# UBER TECHNOLOGIES, INC.\\n\\n(Exact name of registrant as specified in its charter)\\n\\nDelaware\\n\\n45-2647441\\n\\n(State or other jurisdiction of incorporation or organization) (I.R.S. Employer Identification No.)\\n\\n1515 3rd Street\\n\\nSan Francisco, California 94158\\n\\n(Address of principal executive offices, including zip code)\\n\\n(415) 612-8582\\n\\n(Registrant’s telephone number, including area code)\\n\\n# Securities registered pursuant to Section 12(b) of the Act:\\n\\n|Title of each class|Trading Symbol(s)|Name of each exchange on which registered|\\n|---|---|---|\\n|Common Stock, par value $0.00001 per share|UBER|New York Stock Exchange|\\n\\nSecurities registered pursuant to Section 12(g) of the Act: None\\n\\nIndicate by check mark whether the registrant is a well-known seasoned issuer, as defined in Rule 405 of the Securities Act. Yes ☒ No ☐\\n\\nIndicate by check mark whether the registrant is not required to file reports pursuant to Section 13 or Section 15(d) of the Act. Yes ☐ No ☒\\n\\nIndicate by check mark whether the registrant (1) has filed all reports required to be filed by Section 13 or 15(d) of the Securities Exchange Act of 1934 during the preceding 12 months (or for such shorter period that the registrant was required to file such reports), and (2) has been subject to such filing requirements for the past 90 days. Yes ☒ No ☐\\n\\nIndicate by check mark whether the registrant has submitted electronically every Interactive Data File required to be submitted pursuant to Rule 405 of Regulation S-T (§232.405 of this chapter) during the preceding 12 months (or for such shorter period that the registrant was required to submit such files). Yes ☒ No ☐\\n\\nIndicate by check mark whether the registrant is a large accelerated filer, an accelerated filer, a non-accelerated filer, a smaller reporting company, or an emerging growth company. See the definitions of “large accelerated filer,” “accelerated filer,” “smaller reporting company,” and “emerging growth company” in Rule 12b-2 of the Exchange Act.', mimetype='text/plain', start_char_idx=None, end_char_idx=None, metadata_seperator='\\n', text_template='{metadata_str}\\n\\n{content}'),\n",
|
||||
" Document(id_='253b1141-a260-466e-b164-b39df67ef799', embedding=None, metadata={}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={}, metadata_template='{key}: {value}', metadata_separator='\\n', text=\"# Large accelerated filer\\n\\n☒\\n\\n# Accelerated filer\\n\\n☐\\n\\n# Non-accelerated filer\\n\\n☐\\n\\n# Smaller reporting company\\n\\n☐\\n\\n# Emerging growth company\\n\\n☐\\n\\nIf an emerging growth company, indicate by check mark if the registrant has elected not to use the extended transition period for complying with any new or revised financial accounting standards provided pursuant to Section 13(a) of the Exchange Act.\\n\\n☐\\n\\nIndicate by check mark whether the registrant has filed a report on and attestation to its management’s assessment of the effectiveness of its internal control over financial reporting under Section 404(b) of the Sarbanes-Oxley Act (15 U.S.C. 7262(b)) by the registered public accounting firm that prepared or issued\\n\\n☒\\n\\nIndicate by check mark whether the registrant is a shell company (as defined in Rule 12b-2 of the Exchange Act). Yes\\n\\n☐\\n\\nNo\\n\\n☒\\n\\nThe aggregate market value of the voting and non-voting common equity held by non-affiliates of the registrant as of June 30, 2021, the last business day of the registrant's most recently completed second fiscal quarter, was approximately $90.5 billion based upon the closing price reported for such date on the New York Stock Exchange.\\n\\nThe number of shares of the registrant's common stock outstanding as of February 22, 2022 was 1,954,464,088.\\n\\n# DOCUMENTS INCORPORATED BY REFERENCE\\n\\nPortions of the registrant’s Definitive Proxy Statement relating to the Annual Meeting of Stockholders are incorporated by reference into Part III of this Annual Report on Form 10-K where indicated. Such Definitive Proxy Statement will be filed with the Securities and Exchange Commission within 120 days after the end of the registrant’s fiscal year ended December 31, 2021.\", mimetype='text/plain', start_char_idx=None, end_char_idx=None, metadata_seperator='\\n', text_template='{metadata_str}\\n\\n{content}'),\n",
|
||||
" Document(id_='ad988239-3ab5-498d-85ba-a29241db24d4', embedding=None, metadata={}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={}, metadata_template='{key}: {value}', metadata_separator='\\n', text='# UBER TECHNOLOGIES, INC.\\n\\n# TABLE OF CONTENTS\\n\\n|Special Note Regarding Forward-Looking Statements|2|\\n|---|---|\\n|PART I|PART I|\\n|Item 1. Business|4|\\n|Item 1A. Risk Factors|11|\\n|Item 1B. Unresolved Staff Comments|46|\\n|Item 2. Properties|46|\\n|Item 3. Legal Proceedings|46|\\n|Item 4. Mine Safety Disclosures|47|\\n|PART II|PART II|\\n|Item 5. Market for Registrant’s Common Equity, Related Stockholder Matters and Issuer Purchases of Equity Securities|47|\\n|Item 6. [Reserved]|48|\\n|Item 7. Management’s Discussion and Analysis of Financial Condition and Results of Operations|48|\\n|Item 7A. Quantitative and Qualitative Disclosures About Market Risk|69|\\n|Item 8. Financial Statements and Supplementary Data|70|\\n|Item 9. Changes in and Disagreements with Accountants on Accounting and Financial Disclosure|146|\\n|Item 9A. Controls and Procedures|147|\\n|Item 9B. Other Information|147|\\n|Item 9C. Disclosure Regarding Foreign Jurisdictions that Prevent Inspections|147|\\n|PART III|PART III|\\n|Item 10. Directors, Executive Officers and Corporate Governance|147|\\n|Item 11. Executive Compensation|147|\\n|Item 12. Security Ownership of Certain Beneficial Owners and Management and Related Stockholder Matters|148|\\n|Item 13. Certain Relationships and Related Transactions, and Director Independence|148|\\n|Item 14. Principal Accounting Fees and Services|148|\\n|PART IV|PART IV|\\n|Item 15. Exhibits, Financial Statement Schedules|148|\\n|Item 16. Form 10-K Summary|148|\\n|Exhibit Index|149|\\n|Signatures|152|', mimetype='text/plain', start_char_idx=None, end_char_idx=None, metadata_seperator='\\n', text_template='{metadata_str}\\n\\n{content}')]"
|
||||
]
|
||||
},
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"documents"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "llamacloud",
|
||||
"language": "python",
|
||||
"name": "llamacloud"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 72 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 173 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 72 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 88 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 200 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 115 KiB |
File diff suppressed because it is too large
Load Diff
Binary file not shown.
|
Before Width: | Height: | Size: 580 KiB |
@@ -11,8 +11,6 @@
|
||||
"\n",
|
||||
"In this cookbook we show you how to build a multimodal report generation agent from a bank of research reports. We use the a set of ICLR papers (which were also used as the dataset in our [DeepLearning.ai course](https://www.deeplearning.ai/short-courses/building-agentic-rag-with-llamaindex/?utm_campaign=llamaindexC2-launch&utm_medium=headband&utm_source=dlai-homepage).\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"We use our workflow abstraction to define an agentic system that contains two main phases: a research phase that pulls in relevant files through chunk-level or file-level retrieval, and then a blog generation phase that synthesizes the final report."
|
||||
]
|
||||
},
|
||||
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 1.5 MiB |
Binary file not shown.
Binary file not shown.
|
Before Width: | Height: | Size: 350 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 47 KiB |
@@ -1,602 +0,0 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"<a href=\"https://colab.research.google.com/github/run-llama/llama_parse/blob/main/examples/parsing_instructions.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
|
||||
"\n",
|
||||
"# Parsing documents with Instructions\n",
|
||||
"\n",
|
||||
"Parsing instructions allow you to guide our parsing model in the same way you would instruct an LLM.\n",
|
||||
"\n",
|
||||
"These instructions can be useful for improving the parser's performance on complex document layouts, extracting data in a specific format, or transforming the document in other ways.\n",
|
||||
"\n",
|
||||
"### Why This Matters:\n",
|
||||
"Traditional document parsing can be rigid and error-prone, often missing crucial context and nuances in complex layouts. Our instruction-based parsing allows you to:\n",
|
||||
"\n",
|
||||
"1. Extract specific information with pinpoint accuracy\n",
|
||||
"2. Handle complex document layouts with ease\n",
|
||||
"3. Transform unstructured data into structured formats effortlessly\n",
|
||||
"4. Save hours of manual data entry and verification\n",
|
||||
"5. Reduce errors in document processing workflows\n",
|
||||
"\n",
|
||||
"In this demonstration, we showcase how parsing instructions can be used to extract specific information from unstructured documents. Below are the documents we use for testing:\n",
|
||||
"\n",
|
||||
"1. McDonald's Receipt - Extracting the price of each order and the final amount to be paid.\n",
|
||||
"\n",
|
||||
"2. Expense Report Document - Extracting employee name, employee ID, position, department, date ranges, individual expense items with dates, categories, and amounts.\n",
|
||||
"\n",
|
||||
"3. Purchase Order Document - Identifying the PO number, vendor details, shipping terms, and an itemized list of products with quantities and unit prices.\n",
|
||||
"\n",
|
||||
"Let's jump into these real-world examples and see how parsing instructions can help us extract specific information."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Installation"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!pip install llama-parse"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Setup API Key"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import nest_asyncio\n",
|
||||
"\n",
|
||||
"nest_asyncio.apply()\n",
|
||||
"\n",
|
||||
"import os\n",
|
||||
"\n",
|
||||
"os.environ[\"LLAMA_CLOUD_API_KEY\"] = \"llx-...\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### McDonald's Receipt\n",
|
||||
"\n",
|
||||
"Here we extract the price of each order and the final amount to be paid."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"<img src=\"mcdonalds_receipt.png\" alt=\"Alt Text\" width=\"500\">"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Started parsing the file under job_id 66643b81-e2f4-408b-890b-8e116472210b\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from llama_parse import LlamaParse\n",
|
||||
"\n",
|
||||
"vanilaParsing = LlamaParse(result_type=\"markdown\").load_data(\"./mcdonalds_receipt.png\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"# Rate us HIGHLY SATISFIED\n",
|
||||
"\n",
|
||||
"Purchase any sandwich and receive a FREE ITEM\n",
|
||||
"\n",
|
||||
"Go to WWW.mcdvoice.com within 7 days of purchase of equal or lesser value and tell us about your visit.\n",
|
||||
"\n",
|
||||
"Validation Code: 31278-01121-21018-20481-00081-0\n",
|
||||
"\n",
|
||||
"Valid at participating US McDonald's\n",
|
||||
"\n",
|
||||
"Expires 30 days after receipt date\n",
|
||||
"\n",
|
||||
"# McDonald's Restaurant #312782378\n",
|
||||
"\n",
|
||||
"PINE RD NW\n",
|
||||
"\n",
|
||||
"RICE MN 56367-9740\n",
|
||||
"\n",
|
||||
"TEL# 320 393 4600\n",
|
||||
"\n",
|
||||
"KS# 12/08/2022 08:48 PM\n",
|
||||
"\n",
|
||||
"# Order\n",
|
||||
"\n",
|
||||
"|Happy Meal 6 Pc|$4.89|\n",
|
||||
"|---|---|\n",
|
||||
"|Creamy Ranch Cup| |\n",
|
||||
"|Extra Kids Fry| |\n",
|
||||
"|Wreck It Ralph 2 Snack| |\n",
|
||||
"|Oreo McFlurry|$2.69|\n",
|
||||
"\n",
|
||||
"# Summary\n",
|
||||
"\n",
|
||||
"|Subtotal|$7.58|\n",
|
||||
"|---|---|\n",
|
||||
"|Tax|$0.52|\n",
|
||||
"|Take-Out Total|$8.10|\n",
|
||||
"|Cash Tendered|$10.00|\n",
|
||||
"|Change|$1.90|\n",
|
||||
"\n",
|
||||
"### Not ACCEPTING APPLICATIONS *++ McDonald's Restaurant Rice\n",
|
||||
"\n",
|
||||
"Text to #36453 apply 31278\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"print(vanilaParsing[0].text)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Started parsing the file under job_id 1a04fdbb-5415-4a36-a1bd-26bfb5d618fa\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"parsingInstruction = \"\"\"The provided document is a McDonald's receipt.\n",
|
||||
" Provide the price of each order and final amount to be paid.\"\"\"\n",
|
||||
"withInstructionParsing = LlamaParse(\n",
|
||||
" result_type=\"markdown\", parsing_instruction=parsingInstruction\n",
|
||||
").load_data(\"./mcdonalds_receipt.png\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Here are the prices for each order from the McDonald's receipt:\n",
|
||||
"\n",
|
||||
"1. Happy Meal 6 Pc: $4.89\n",
|
||||
"2. Snack Oreo McFlurry: $2.69\n",
|
||||
"\n",
|
||||
"**Subtotal:** $7.58\n",
|
||||
"**Tax:** $0.52\n",
|
||||
"**Total Amount to be Paid:** $8.10\n",
|
||||
"\n",
|
||||
"The cash tendered was $10.00, and the change given was $1.90.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"print(withInstructionParsing[0].text)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Expense Report Document\n",
|
||||
"\n",
|
||||
"Here we extract employee name, employee ID, position, department, date ranges, individual expense items with dates, categories, and amounts."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"<img src=\"expense_report_document.png\" alt=\"Alt Text\" width=\"500\">"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Started parsing the file under job_id b6bcc6e1-7d30-4522-9abd-ace196781a70\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"vanilaParsing = LlamaParse(result_type=\"markdown\").load_data(\n",
|
||||
" \"./expense_report_document.pdf\"\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"# QUANTUM DYNAMICS CORPORATION\n",
|
||||
"\n",
|
||||
"# EMPLOYEE EXPENSE REPORT\n",
|
||||
"\n",
|
||||
"# FISCAL YEAR 2024\n",
|
||||
"\n",
|
||||
"# EMPLOYEE INFORMATION:\n",
|
||||
"\n",
|
||||
"Name: Dr. Alexandra Chen-Martinez, PhD\n",
|
||||
"\n",
|
||||
"Employee ID: QD-2022-1457\n",
|
||||
"\n",
|
||||
"Department: Advanced Research & Development\n",
|
||||
"\n",
|
||||
"Cost Center: CC-ARD-NA-003\n",
|
||||
"\n",
|
||||
"Project Codes: QD-QUANTUM-2024-01, QD-AI-2024-03\n",
|
||||
"\n",
|
||||
"Position: Principal Research Scientist\n",
|
||||
"\n",
|
||||
"Reporting Manager: Dr. James Thompson\n",
|
||||
"\n",
|
||||
"# TRIP/EXPENSE PERIOD:\n",
|
||||
"\n",
|
||||
"Start Date: November 15, 2024\n",
|
||||
"\n",
|
||||
"End Date: December 10, 2024\n",
|
||||
"\n",
|
||||
"Purpose: International Conference Attendance & Client Meetings\n",
|
||||
"\n",
|
||||
"Locations: Tokyo, Japan → Singapore → Sydney, Australia\n",
|
||||
"\n",
|
||||
"# CURRENCY CONVERSION RATES APPLIED:\n",
|
||||
"\n",
|
||||
"JPY (¥) → USD: 0.0068 (as of 11/15/2024)\n",
|
||||
"\n",
|
||||
"SGD (S$) → USD: 0.74 (as of 11/28/2024)\n",
|
||||
"\n",
|
||||
"AUD (A$) → USD: 0.65 (as of 12/03/2024)\n",
|
||||
"\n",
|
||||
"# ITEMIZED EXPENSES:\n",
|
||||
"\n",
|
||||
"|Date|Category|Description|Original|Currency|USD|\n",
|
||||
"|---|---|---|---|---|---|\n",
|
||||
"|11/15/2024|Transportation|JFK → NRT Business Class|4,250.00|USD|4,250.00|\n",
|
||||
"|Booking Ref: QF78956 - Corporate Rate Applied|Booking Ref: QF78956 - Corporate Rate Applied|Booking Ref: QF78956 - Corporate Rate Applied|Booking Ref: QF78956 - Corporate Rate Applied|Booking Ref: QF78956 - Corporate Rate Applied|Booking Ref: QF78956 - Corporate Rate Applied|\n",
|
||||
"|Project Code: QD-QUANTUM-2024-01|Project Code: QD-QUANTUM-2024-01|Project Code: QD-QUANTUM-2024-01|Project Code: QD-QUANTUM-2024-01|Project Code: QD-QUANTUM-2024-01|Project Code: QD-QUANTUM-2024-01|\n",
|
||||
"|11/16/2024|Accommodation|Hilton Tokyo - 5 nights|225,000|JPY|1,530.00|\n",
|
||||
"|Confirmation: HTK-2024-78956|Confirmation: HTK-2024-78956|Confirmation: HTK-2024-78956|Confirmation: HTK-2024-78956|Confirmation: HTK-2024-78956|Confirmation: HTK-2024-78956|\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"print(vanilaParsing[0].text)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Started parsing the file under job_id 7b0d05bb-947b-4475-8d0f-f10386f7446e\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"parsingInstruction = \"\"\"You are provided with an expense report. \n",
|
||||
"Extract employee name, employee id, position, department, date ranges, individual expense items with dates, categories, and amounts.\"\"\"\n",
|
||||
"\n",
|
||||
"withInstructionParsing = LlamaParse(\n",
|
||||
" result_type=\"markdown\", parsing_instruction=parsingInstruction\n",
|
||||
").load_data(\"./expense_report_document.pdf\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"**Employee Information:**\n",
|
||||
"- **Name:** Dr. Alexandra Chen-Martinez, PhD\n",
|
||||
"- **Employee ID:** QD-2022-1457\n",
|
||||
"- **Position:** Principal Research Scientist\n",
|
||||
"- **Department:** Advanced Research & Development\n",
|
||||
"\n",
|
||||
"**Trip/Expense Period:**\n",
|
||||
"- **Start Date:** November 15, 2024\n",
|
||||
"- **End Date:** December 10, 2024\n",
|
||||
"\n",
|
||||
"**Expense Items:**\n",
|
||||
"1. **Date:** 11/15/2024\n",
|
||||
"- **Category:** Transportation\n",
|
||||
"- **Description:** JFK → NRT Business Class\n",
|
||||
"- **Original Amount:** $4,250.00\n",
|
||||
"- **Currency:** USD\n",
|
||||
"- **USD Amount:** $4,250.00\n",
|
||||
"- **Booking Reference:** QF78956 - Corporate Rate Applied\n",
|
||||
"- **Project Code:** QD-QUANTUM-2024-01\n",
|
||||
"\n",
|
||||
"2. **Date:** 11/16/2024\n",
|
||||
"- **Category:** Accommodation\n",
|
||||
"- **Description:** Hilton Tokyo - 5 nights\n",
|
||||
"- **Original Amount:** ¥225,000\n",
|
||||
"- **Currency:** JPY\n",
|
||||
"- **USD Amount:** $1,530.00\n",
|
||||
"- **Confirmation:** HTK-2024-78956\n",
|
||||
"\n",
|
||||
"**Locations:**\n",
|
||||
"- Tokyo, Japan\n",
|
||||
"- Singapore\n",
|
||||
"- Sydney, Australia\n",
|
||||
"\n",
|
||||
"**Currency Conversion Rates Applied:**\n",
|
||||
"- JPY (¥) → USD: 0.0068 (as of 11/15/2024)\n",
|
||||
"- SGD (S$) → USD: 0.74 (as of 11/28/2024)\n",
|
||||
"- AUD (A$) → USD: 0.65 (as of 12/03/2024)\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"print(withInstructionParsing[0].text)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Purchase Order Document \n",
|
||||
"\n",
|
||||
"Here we identify the PO number, vendor details, shipping terms, and an itemized list of products with quantities and unit prices."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"<img src=\"purchase_order_document.png\" alt=\"Alt Text\" width=\"500\">"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Started parsing the file under job_id b8cb11c3-7dce-4e6a-94bb-1a4e50e45e55\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"vanilaParsing = LlamaParse(result_type=\"markdown\").load_data(\n",
|
||||
" \"./purchase_order_document.pdf\"\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"# GLOBAL TECH SOLUTIONS, INC.\n",
|
||||
"\n",
|
||||
"# PURCHASE ORDER\n",
|
||||
"\n",
|
||||
"Document Reference: PO-2024-GT-9876/REV.2\n",
|
||||
"\n",
|
||||
"[Original: PO-2024-GT-9876]\n",
|
||||
"\n",
|
||||
"Amendment Date: 12/10/2024\n",
|
||||
"\n",
|
||||
"# VENDOR INFORMATION:\n",
|
||||
"\n",
|
||||
"Quantum Electronics Manufacturing\n",
|
||||
"\n",
|
||||
"DUNS: 78-456-7890\n",
|
||||
"\n",
|
||||
"Tax ID: EU8976543210\n",
|
||||
"\n",
|
||||
"Hoofdorp, Netherlands\n",
|
||||
"\n",
|
||||
"Vendor #: QEM-EU-2024-001\n",
|
||||
"\n",
|
||||
"# SHIP TO:\n",
|
||||
"\n",
|
||||
"Global Tech Solutions, Inc.\n",
|
||||
"\n",
|
||||
"Building 7A, Innovation Park\n",
|
||||
"\n",
|
||||
"2100 Technology Drive\n",
|
||||
"\n",
|
||||
"Austin, TX 78701\n",
|
||||
"\n",
|
||||
"USA\n",
|
||||
"\n",
|
||||
"Attn: Sarah Martinez, Receiving Manager\n",
|
||||
"\n",
|
||||
"Tel: +1 (512) 555-0123\n",
|
||||
"\n",
|
||||
"# PAYMENT TERMS:\n",
|
||||
"\n",
|
||||
"Net 45\n",
|
||||
"\n",
|
||||
"2% discount if paid within 15 days\n",
|
||||
"\n",
|
||||
"# SHIPPING TERMS:\n",
|
||||
"\n",
|
||||
"DDP (Delivered Duty Paid) - Incoterms 2020\n",
|
||||
"\n",
|
||||
"Insurance Required: Yes\n",
|
||||
"\n",
|
||||
"Preferred Carrier: DHL/FedEx\n",
|
||||
"\n",
|
||||
"Required Delivery Date: 01/15/2025\n",
|
||||
"\n",
|
||||
"# SPECIAL INSTRUCTIONS:\n",
|
||||
"\n",
|
||||
"1. All shipments must include Certificate of Conformance\n",
|
||||
"2. ESD-sensitive items must be properly packaged\n",
|
||||
"3. Temperature logging required for items marked with *\n",
|
||||
"4. Partial shipments accepted with prior approval\n",
|
||||
"5. Quote PO number on all correspondence\n",
|
||||
"\n",
|
||||
"# ITEM DETAILS:\n",
|
||||
"\n",
|
||||
"|Line|Part Number|Description|Qty|UOM|Unit Price|Total|\n",
|
||||
"|---|---|---|---|---|---|---|\n",
|
||||
"|1|QE-MCU-5590|Microcontroller Unit|500|EA|$12.50|$6,250.00|\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"print(vanilaParsing[0].text)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Started parsing the file under job_id d2731305-984d-4633-8a52-0493748cf10b\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"parsingInstruction = \"\"\"You are provided with a purchase order. \n",
|
||||
"Identify the PO number, vendor details, shipping terms, and itemized list of products with quantities and unit prices.\"\"\"\n",
|
||||
"\n",
|
||||
"withInstructionParsing = LlamaParse(\n",
|
||||
" result_type=\"markdown\", parsing_instruction=parsingInstruction\n",
|
||||
").load_data(\"./purchase_order_document.pdf\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Here are the details extracted from the purchase order:\n",
|
||||
"\n",
|
||||
"**PO Number:** PO-2024-GT-9876/REV.2\n",
|
||||
"\n",
|
||||
"**Vendor Details:**\n",
|
||||
"- **Vendor Name:** Quantum Electronics Manufacturing\n",
|
||||
"- **DUNS:** 78-456-7890\n",
|
||||
"- **Tax ID:** EU8976543210\n",
|
||||
"- **Address:** Hoofdorp, Netherlands\n",
|
||||
"- **Vendor Number:** QEM-EU-2024-001\n",
|
||||
"- **Contact Person:** Sarah Martinez, Receiving Manager\n",
|
||||
"- **Phone:** +1 (512) 555-0123\n",
|
||||
"\n",
|
||||
"**Shipping Terms:**\n",
|
||||
"- **Terms:** DDP (Delivered Duty Paid) - Incoterms 2020\n",
|
||||
"- **Insurance Required:** Yes\n",
|
||||
"- **Preferred Carrier:** DHL/FedEx\n",
|
||||
"- **Required Delivery Date:** 01/15/2025\n",
|
||||
"\n",
|
||||
"**Itemized List of Products:**\n",
|
||||
"1. **Part Number:** QE-MCU-5590\n",
|
||||
"- **Description:** Microcontroller Unit\n",
|
||||
"- **Quantity:** 500 EA\n",
|
||||
"- **Unit Price:** $12.50\n",
|
||||
"- **Total:** $6,250.00\n",
|
||||
"\n",
|
||||
"**Payment Terms:**\n",
|
||||
"- Net 45\n",
|
||||
"- 2% discount if paid within 15 days\n",
|
||||
"\n",
|
||||
"**Special Instructions:**\n",
|
||||
"1. All shipments must include Certificate of Conformance\n",
|
||||
"2. ESD-sensitive items must be properly packaged\n",
|
||||
"3. Temperature logging required for items marked with *\n",
|
||||
"4. Partial shipments accepted with prior approval\n",
|
||||
"5. Quote PO number on all correspondence\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"print(withInstructionParsing[0].text)"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "llamacloud",
|
||||
"language": "python",
|
||||
"name": "llamacloud"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
Binary file not shown.
Binary file not shown.
|
Before Width: | Height: | Size: 344 KiB |
File diff suppressed because it is too large
Load Diff
Binary file not shown.
|
Before Width: | Height: | Size: 2.3 MiB |
Binary file not shown.
|
Before Width: | Height: | Size: 100 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 464 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 410 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 444 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 610 KiB |
File diff suppressed because it is too large
Load Diff
Binary file not shown.
|
Before Width: | Height: | Size: 986 KiB |
@@ -46,7 +46,7 @@
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"os.environ[\"LLAMA_CLOUD_API_KEY\"] = \"<LLAMA_CLOUD_API_KEY>\""
|
||||
"os.environ[\"LLAMA_CLOUD_API_KEY\"] = \"<LLAMA_CLOUD_API_KEY>"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
+203
-721
File diff suppressed because it is too large
Load Diff
@@ -1,92 +0,0 @@
|
||||
import click
|
||||
import json
|
||||
from enum import Enum
|
||||
from pathlib import Path
|
||||
from pydantic.fields import FieldInfo
|
||||
from typing import Any, Callable, List
|
||||
|
||||
from llama_parse.base import LlamaParse
|
||||
|
||||
|
||||
def pydantic_field_to_click_option(name: str, field: FieldInfo) -> click.Option:
|
||||
"""Convert a Pydantic field to a Click option."""
|
||||
kwargs = {
|
||||
"default": field.default if field.default else None,
|
||||
"help": field.description,
|
||||
}
|
||||
|
||||
if isinstance(kwargs["default"], Enum):
|
||||
kwargs["default"] = kwargs["default"].value
|
||||
|
||||
if field.annotation is bool:
|
||||
kwargs["is_flag"] = True
|
||||
if field.default and field.default is True:
|
||||
name = f"no-{name}"
|
||||
return click.option(f'--{name.replace("_", "-")}', **kwargs)
|
||||
|
||||
|
||||
def add_options(options: List[click.Option]) -> Callable:
|
||||
def _add_options(func: Callable) -> Callable:
|
||||
for option in reversed(options):
|
||||
func = option(func)
|
||||
return func
|
||||
|
||||
return _add_options
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.argument("file_paths", nargs=-1, type=click.Path(exists=True, path_type=Path))
|
||||
@click.option(
|
||||
"--output-file", type=click.Path(path_type=Path), help="Path to save the output"
|
||||
)
|
||||
@click.option("--output-raw-json", is_flag=True, help="Output the raw JSON result")
|
||||
@add_options(
|
||||
[
|
||||
pydantic_field_to_click_option(name, field)
|
||||
for name, field in LlamaParse.model_fields.items()
|
||||
if name not in ["custom_client"]
|
||||
]
|
||||
)
|
||||
def parse(**kwargs: Any) -> None:
|
||||
"""Parse files using LlamaParse and output the results."""
|
||||
file_paths = kwargs.pop("file_paths")
|
||||
output_file = kwargs.pop("output_file")
|
||||
output_raw_json = kwargs.pop("output_raw_json")
|
||||
|
||||
# Remove None values to use LlamaParse defaults
|
||||
kwargs = {k: v for k, v in kwargs.items() if v is not None}
|
||||
|
||||
# Remove no- prefix for boolean flags
|
||||
kwargs = {k.replace("no_", ""): v for k, v in kwargs.items()}
|
||||
|
||||
parser = LlamaParse(**kwargs)
|
||||
if output_raw_json:
|
||||
results = parser.get_json_result(list(file_paths))
|
||||
|
||||
if output_file:
|
||||
with output_file.open("w") as f:
|
||||
json.dump(results, f)
|
||||
click.echo(f"Results saved to {output_file}")
|
||||
else:
|
||||
click.echo(results)
|
||||
else:
|
||||
results = parser.load_data(list(file_paths))
|
||||
|
||||
if output_file:
|
||||
with output_file.open("w") as f:
|
||||
for i, doc in enumerate(results):
|
||||
f.write(f"File: {doc.metadata.get('file_path', 'Unknown')}\n") # type: ignore
|
||||
f.write(doc.text) # type: ignore
|
||||
if i < len(results) - 1:
|
||||
f.write("\n\n---\n\n")
|
||||
click.echo(f"Results saved to {output_file}")
|
||||
else:
|
||||
for i, doc in enumerate(results):
|
||||
click.echo(f"File: {doc.metadata.get('file_path', 'Unknown')}") # type: ignore
|
||||
click.echo(doc.text) # type: ignore
|
||||
if i < len(results) - 1:
|
||||
click.echo("\n---\n")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
parse()
|
||||
@@ -10,8 +10,6 @@ class ResultType(str, Enum):
|
||||
|
||||
TXT = "text"
|
||||
MD = "markdown"
|
||||
JSON = "json"
|
||||
STRUCTURED = "structured"
|
||||
|
||||
|
||||
class Language(str, Enum):
|
||||
@@ -191,11 +189,4 @@ SUPPORTED_FILE_TYPES = [
|
||||
".xlr",
|
||||
".eth",
|
||||
".tsv",
|
||||
".mp3",
|
||||
".mp4",
|
||||
".mpeg",
|
||||
".mpga",
|
||||
".m4a",
|
||||
".wav",
|
||||
".webm",
|
||||
]
|
||||
|
||||
Generated
+1248
-1472
File diff suppressed because it is too large
Load Diff
+2
-8
@@ -4,7 +4,7 @@ build-backend = "poetry.core.masonry.api"
|
||||
|
||||
[tool.poetry]
|
||||
name = "llama-parse"
|
||||
version = "0.5.20"
|
||||
version = "0.5.7"
|
||||
description = "Parse files into RAG-Optimized formats."
|
||||
authors = ["Logan Markewich <logan@llamaindex.ai>"]
|
||||
license = "MIT"
|
||||
@@ -12,15 +12,9 @@ readme = "README.md"
|
||||
packages = [{include = "llama_parse"}]
|
||||
|
||||
[tool.poetry.dependencies]
|
||||
python = ">=3.9,<4.0"
|
||||
python = ">=3.8.1,<4.0"
|
||||
llama-index-core = ">=0.11.0"
|
||||
pydantic = "!=2.10"
|
||||
click = "^8.1.7"
|
||||
|
||||
[tool.poetry.group.dev.dependencies]
|
||||
pytest = "^8.0.0"
|
||||
pytest-asyncio = "*"
|
||||
ipykernel = "^6.29.0"
|
||||
|
||||
[tool.poetry.scripts]
|
||||
llama-parse = "llama_parse.cli.main:parse"
|
||||
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 347 KiB |
+7
-77
@@ -1,6 +1,5 @@
|
||||
import os
|
||||
import pytest
|
||||
import shutil
|
||||
from fsspec.implementations.local import LocalFileSystem
|
||||
from httpx import AsyncClient
|
||||
|
||||
@@ -77,14 +76,13 @@ def test_simple_page_markdown_buffer(markdown_parser: LlamaParse) -> None:
|
||||
os.environ.get("LLAMA_CLOUD_API_KEY", "") == "",
|
||||
reason="LLAMA_CLOUD_API_KEY not set",
|
||||
)
|
||||
@pytest.mark.asyncio
|
||||
async def test_simple_page_with_custom_fs() -> None:
|
||||
def test_simple_page_with_custom_fs() -> None:
|
||||
parser = LlamaParse(result_type="markdown")
|
||||
fs = LocalFileSystem()
|
||||
filepath = os.path.join(
|
||||
os.path.dirname(__file__), "test_files/attention_is_all_you_need.pdf"
|
||||
)
|
||||
result = await parser.aload_data(filepath, fs=fs)
|
||||
result = parser.load_data(filepath, fs=fs)
|
||||
assert len(result) == 1
|
||||
|
||||
|
||||
@@ -92,14 +90,13 @@ async def test_simple_page_with_custom_fs() -> None:
|
||||
os.environ.get("LLAMA_CLOUD_API_KEY", "") == "",
|
||||
reason="LLAMA_CLOUD_API_KEY not set",
|
||||
)
|
||||
@pytest.mark.asyncio
|
||||
async def test_simple_page_progress_workers() -> None:
|
||||
def test_simple_page_progress_workers() -> None:
|
||||
parser = LlamaParse(result_type="markdown", show_progress=True, verbose=True)
|
||||
|
||||
filepath = os.path.join(
|
||||
os.path.dirname(__file__), "test_files/attention_is_all_you_need.pdf"
|
||||
)
|
||||
result = await parser.aload_data([filepath, filepath])
|
||||
result = parser.load_data([filepath, filepath])
|
||||
assert len(result) == 2
|
||||
assert len(result[0].text) > 0
|
||||
|
||||
@@ -110,7 +107,7 @@ async def test_simple_page_progress_workers() -> None:
|
||||
filepath = os.path.join(
|
||||
os.path.dirname(__file__), "test_files/attention_is_all_you_need.pdf"
|
||||
)
|
||||
result = await parser.aload_data([filepath, filepath])
|
||||
result = parser.load_data([filepath, filepath])
|
||||
assert len(result) == 2
|
||||
assert len(result[0].text) > 0
|
||||
|
||||
@@ -119,79 +116,12 @@ async def test_simple_page_progress_workers() -> None:
|
||||
os.environ.get("LLAMA_CLOUD_API_KEY", "") == "",
|
||||
reason="LLAMA_CLOUD_API_KEY not set",
|
||||
)
|
||||
@pytest.mark.asyncio
|
||||
async def test_custom_client() -> None:
|
||||
def test_custom_client() -> None:
|
||||
custom_client = AsyncClient(verify=False, timeout=10)
|
||||
parser = LlamaParse(result_type="markdown", custom_client=custom_client)
|
||||
filepath = os.path.join(
|
||||
os.path.dirname(__file__), "test_files/attention_is_all_you_need.pdf"
|
||||
)
|
||||
result = await parser.aload_data(filepath)
|
||||
result = parser.load_data(filepath)
|
||||
assert len(result) == 1
|
||||
assert len(result[0].text) > 0
|
||||
|
||||
|
||||
@pytest.mark.skipif(
|
||||
os.environ.get("LLAMA_CLOUD_API_KEY", "") == "",
|
||||
reason="LLAMA_CLOUD_API_KEY not set",
|
||||
)
|
||||
@pytest.mark.asyncio
|
||||
async def test_input_url() -> None:
|
||||
parser = LlamaParse(result_type="markdown")
|
||||
|
||||
# links to a resume example
|
||||
input_url = "https://cdn-blog.novoresume.com/articles/google-docs-resume-templates/basic-google-docs-resume.png"
|
||||
result = await parser.aload_data(input_url)
|
||||
assert len(result) == 1
|
||||
assert "your name" in result[0].text.lower()
|
||||
|
||||
|
||||
@pytest.mark.skipif(
|
||||
os.environ.get("LLAMA_CLOUD_API_KEY", "") == "",
|
||||
reason="LLAMA_CLOUD_API_KEY not set",
|
||||
)
|
||||
@pytest.mark.asyncio
|
||||
async def test_input_url_with_website_input() -> None:
|
||||
parser = LlamaParse(result_type="markdown")
|
||||
input_url = "https://www.example.com"
|
||||
result = await parser.aload_data(input_url)
|
||||
assert len(result) == 1
|
||||
assert "example" in result[0].text.lower()
|
||||
|
||||
|
||||
@pytest.mark.skipif(
|
||||
os.environ.get("LLAMA_CLOUD_API_KEY", "") == "",
|
||||
reason="LLAMA_CLOUD_API_KEY not set",
|
||||
)
|
||||
@pytest.mark.asyncio
|
||||
async def test_mixing_input_types() -> None:
|
||||
parser = LlamaParse(result_type="markdown")
|
||||
filepath = os.path.join(
|
||||
os.path.dirname(__file__), "test_files/attention_is_all_you_need.pdf"
|
||||
)
|
||||
input_url = "https://cdn-blog.novoresume.com/articles/google-docs-resume-templates/basic-google-docs-resume.png"
|
||||
result = await parser.aload_data([filepath, input_url])
|
||||
|
||||
assert len(result) == 2
|
||||
|
||||
|
||||
@pytest.mark.skipif(
|
||||
os.environ.get("LLAMA_CLOUD_API_KEY", "") == "",
|
||||
reason="LLAMA_CLOUD_API_KEY not set",
|
||||
)
|
||||
@pytest.mark.asyncio
|
||||
async def test_download_images() -> None:
|
||||
parser = LlamaParse(result_type="markdown", take_screenshot=True)
|
||||
filepath = os.path.join(
|
||||
os.path.dirname(__file__), "test_files/attention_is_all_you_need.pdf"
|
||||
)
|
||||
json_result = await parser.aget_json([filepath])
|
||||
|
||||
assert len(json_result) == 1
|
||||
assert len(json_result[0]["pages"][0]["images"]) > 0
|
||||
|
||||
download_path = os.path.join(os.path.dirname(__file__), "test_files/images")
|
||||
shutil.rmtree(download_path, ignore_errors=True)
|
||||
|
||||
await parser.aget_images(json_result, download_path)
|
||||
assert len(os.listdir(download_path)) == len(json_result[0]["pages"][0]["images"])
|
||||
|
||||
Reference in New Issue
Block a user