mirror of
https://github.com/run-llama/llama_cloud_services.git
synced 2026-07-21 03:55:22 -04:00
Compare commits
14 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Pierre-Loic Doulcet
|
963bd2dd2f | ||
|
Ravi Theja
|
cdbddef86d | ||
|
Pierre-Loic Doulcet
|
3690109abf | ||
|
Pierre-Loic Doulcet
|
2e322b4fc8 | ||
|
Pierre-Loic Doulcet
|
735e5f3ddc | ||
|
Logan
|
e4cb4c75e5 | ||
|
Jerry Liu
|
1693deff72 | ||
|
Jerry Liu
|
3270f1228d | ||
|
Pierre-Loic Doulcet
|
eeabf48d29 | ||
|
Pierre-Loic Doulcet
|
89348aa8e5 | ||
|
Thiago Salvatore
|
3ab2ce27b5 | ||
|
Sacha Bron
|
265261862f | ||
|
Sacha Bron
|
66cf052b8c | ||
|
Jerry Liu
|
2ca2d81e58 |
@@ -7,8 +7,6 @@ assignees: ''
|
||||
|
||||
---
|
||||
|
||||
_Note: we're aware of some missing content in the output and layout issues on tables. Please refrain from opening new issues on this topic unless if you think it's different from what has already been reported._
|
||||
|
||||
**Describe the bug**
|
||||
Write a concise description of what the bug is.
|
||||
|
||||
@@ -19,19 +17,15 @@ If possible, please provide the PDF file causing the issue.
|
||||
If you have it, please provide the ID of the job you ran.
|
||||
You can find it here: https://cloud.llamaindex.ai/parse in the "History" tab.
|
||||
|
||||
**Screenshots**
|
||||
Feel free to also provide screenshots if relevant.
|
||||
|
||||
**Client:**
|
||||
Please remove untested options:
|
||||
- Frontend (cloud.llamaindex.ai)
|
||||
- Python Library
|
||||
- API
|
||||
- Frontend (cloud.llamaindex.ai)
|
||||
- Typescript Library
|
||||
- Notebook
|
||||
- API
|
||||
|
||||
**Options**
|
||||
What options did you use? Multimodal, fast mode, parsing instructions, etc.
|
||||
|
||||
**Additional context**
|
||||
Add any additional context about the problem here.
|
||||
What options did you use? Premium mode, multimodal, fast mode, parsing instructions, etc.
|
||||
Screenshots, code snippets, etc.
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
Binary file not shown.
|
After Width: | Height: | Size: 6.9 MiB |
@@ -0,0 +1,415 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "97c79c38-38a3-40f3-ba2e-250649347d63",
|
||||
"metadata": {
|
||||
"id": "97c79c38-38a3-40f3-ba2e-250649347d63"
|
||||
},
|
||||
"source": [
|
||||
"<a href=\"https://colab.research.google.com/github/run-llama/llama_parse/blob/main/examples/demo_starter_multimodal.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "4e081457",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Multimodal Parsing using LlamaParse\n",
|
||||
"\n",
|
||||
"This cookbook shows you how to use LlamaParse to parse any document with the multimodal capabilities of Multi-Modal LLMs from Anthropic/ OpenAI.\n",
|
||||
"\n",
|
||||
"LlamaParse allows you to plug in external, multimodal model vendors for parsing - we handle the error correction, validation, and scalability/reliability for you.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "qOdqBxCS51Ow",
|
||||
"metadata": {
|
||||
"id": "qOdqBxCS51Ow"
|
||||
},
|
||||
"source": [
|
||||
"### Installation"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "H_Vqcylb50vm",
|
||||
"metadata": {
|
||||
"id": "H_Vqcylb50vm"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!pip install llama-parse"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "15e60ecf-519c-41fc-911b-765adaf8bad4",
|
||||
"metadata": {
|
||||
"id": "15e60ecf-519c-41fc-911b-765adaf8bad4"
|
||||
},
|
||||
"source": [
|
||||
"### Setup\n",
|
||||
"\n",
|
||||
"Here we setup `LLAMA_CLOUD_API_KEY` for using `LlamaParse`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"id": "91a9e532-1454-40e0-bbf0-fd442c350121",
|
||||
"metadata": {
|
||||
"id": "91a9e532-1454-40e0-bbf0-fd442c350121"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import nest_asyncio\n",
|
||||
"\n",
|
||||
"nest_asyncio.apply()\n",
|
||||
"\n",
|
||||
"import os\n",
|
||||
"\n",
|
||||
"# API access to llama-cloud\n",
|
||||
"os.environ[\"LLAMA_CLOUD_API_KEY\"] = \"<YOUR LLAMACLOUD API KEY>\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "LGwBNPNotZRQ",
|
||||
"metadata": {
|
||||
"id": "LGwBNPNotZRQ"
|
||||
},
|
||||
"source": [
|
||||
"## Download Data\n",
|
||||
"\n",
|
||||
"For this demonstration, we will use OpenAI's recent paper `Evaluation of OpenAI o1: Opportunities and Challenges of AGI`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "IjtKDQRLrylI",
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "IjtKDQRLrylI",
|
||||
"outputId": "31df0fac-51f2-4697-f78b-0b7c0b8cd145"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"--2024-12-05 18:54:24-- https://arxiv.org/pdf/2409.18486\n",
|
||||
"Resolving arxiv.org (arxiv.org)... 151.101.67.42, 151.101.131.42, 151.101.3.42, ...\n",
|
||||
"Connecting to arxiv.org (arxiv.org)|151.101.67.42|:443... connected.\n",
|
||||
"HTTP request sent, awaiting response... 200 OK\n",
|
||||
"Length: 13986265 (13M) [application/pdf]\n",
|
||||
"Saving to: ‘o1.pdf’\n",
|
||||
"\n",
|
||||
"o1.pdf 100%[===================>] 13.34M 11.8MB/s in 1.1s \n",
|
||||
"\n",
|
||||
"2024-12-05 18:54:26 (11.8 MB/s) - ‘o1.pdf’ saved [13986265/13986265]\n",
|
||||
"\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"!wget \"https://arxiv.org/pdf/2409.18486\" -O \"o1.pdf\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "4e29a9d7-5bd9-4fb8-8ec1-4c128a748662",
|
||||
"metadata": {
|
||||
"id": "4e29a9d7-5bd9-4fb8-8ec1-4c128a748662"
|
||||
},
|
||||
"source": [
|
||||
"## Initialize LlamaParse\n",
|
||||
"\n",
|
||||
"Initialize LlamaParse in multimodal mode, and specify the vendor.\n",
|
||||
"\n",
|
||||
"**NOTE**: optionally you can specify the Anthropic/ OpenAI API key. If you choose to do so LlamaParse will only charge you 1 credit (0.3c) per page. \n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Using your own API key may incur additional costs from your model provider and could result in failed pages or documents if you do not have sufficient usage limits."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"id": "dc921729-3446-42ca-8e1b-a6fd26195ed9",
|
||||
"metadata": {
|
||||
"id": "dc921729-3446-42ca-8e1b-a6fd26195ed9"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from llama_index.core.schema import TextNode\n",
|
||||
"from typing import List\n",
|
||||
"\n",
|
||||
"def get_text_nodes(json_list: List[dict]):\n",
|
||||
" text_nodes = []\n",
|
||||
" for idx, page in enumerate(json_list):\n",
|
||||
" text_node = TextNode(text=page[\"md\"], metadata={\"page\": page[\"page\"]})\n",
|
||||
" text_nodes.append(text_node)\n",
|
||||
" return text_nodes"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "1b5d6da6",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### With anthropic-sonnet-3.5"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"id": "f2e9d9cf-8189-4fcb-b34f-cde6cc0b59c8",
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "f2e9d9cf-8189-4fcb-b34f-cde6cc0b59c8",
|
||||
"outputId": "a337cbdd-60db-4a73-b66b-2bd6159e81f2"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Started parsing the file under job_id dd9d5e0f-160e-486a-89a2-6005e5a1c2ac\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from llama_parse import LlamaParse\n",
|
||||
"\n",
|
||||
"parser = LlamaParse(\n",
|
||||
" result_type=\"markdown\",\n",
|
||||
" use_vendor_multimodal_model=True,\n",
|
||||
" vendor_multimodal_model_name=\"anthropic-sonnet-3.5\",\n",
|
||||
" target_pages=\"24\"\n",
|
||||
" # invalidate_cache=True\n",
|
||||
")\n",
|
||||
"json_objs = parser.get_json_result(\"o1.pdf\")\n",
|
||||
"json_list = json_objs[0][\"pages\"]\n",
|
||||
"docs = get_text_nodes(json_list)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "4f3c51b0-7878-48d7-9bc3-02b516500128",
|
||||
"metadata": {
|
||||
"id": "4f3c51b0-7878-48d7-9bc3-02b516500128"
|
||||
},
|
||||
"source": [
|
||||
"### With GPT-4o\n",
|
||||
"\n",
|
||||
"For comparison, we will also parse the document using GPT-4o."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"id": "6fc3f258-50ae-4988-b904-c105463a498f",
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "6fc3f258-50ae-4988-b904-c105463a498f",
|
||||
"outputId": "89c525c4-2b93-4909-9657-55646e034637"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Started parsing the file under job_id 6a4dea44-4f90-406b-b290-9e98620b1232\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from llama_parse import LlamaParse\n",
|
||||
"\n",
|
||||
"parser_gpt4o = LlamaParse(\n",
|
||||
" result_type=\"markdown\",\n",
|
||||
" use_vendor_multimodal_model=True,\n",
|
||||
" vendor_multimodal_model=\"openai-gpt4o\",\n",
|
||||
" target_pages=\"24\",\n",
|
||||
" # invalidate_cache=True\n",
|
||||
")\n",
|
||||
"json_objs_gpt4o = parser_gpt4o.get_json_result(\"o1.pdf\")\n",
|
||||
"json_list_gpt4o = json_objs_gpt4o[0][\"pages\"]\n",
|
||||
"docs_gpt4o = get_text_nodes(json_list_gpt4o)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "44c20f7a-2901-4dd0-b635-a4b33c5664c1",
|
||||
"metadata": {
|
||||
"id": "44c20f7a-2901-4dd0-b635-a4b33c5664c1"
|
||||
},
|
||||
"source": [
|
||||
"### View Results\n",
|
||||
"\n",
|
||||
"Let's visualize the results along with the original document page."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"id": "778698aa-da7e-4081-b3b5-0372f228536f",
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "778698aa-da7e-4081-b3b5-0372f228536f",
|
||||
"outputId": "bb89e323-7041-4fc3-d835-95e373189d02"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"page: 25\n",
|
||||
"\n",
|
||||
"| Participant_ID | clinical Description Reference |\n",
|
||||
"|-----------------|----------------------------------|\n",
|
||||
"| Attribute | Value | Basic Personal Information: Subject 098_S_0896 is a 72.0-year-old Female who has completed 15 years of education. The ethnicity is Not Hisp/Latino and race is White. Marital status is Married. Initially diagnosed as AD, as of the date 2007-10-24, the final diagnosis was Dementia. |\n",
|
||||
"| Age | 72.0 |\n",
|
||||
"| Sex | Female |\n",
|
||||
"| Education | 15 |\n",
|
||||
"| Race | White | Biomarker Measurements: The subject's genetic profile includes an ApoE4 status of 0.0... |\n",
|
||||
"| DX_bl | AD |\n",
|
||||
"| DX | Dementia |\n",
|
||||
"| ... | ... | Cognitive and Neurofunctional Assessments: The Mini-Mental State Examination score stands at 29.0. The Clinical Dementia Rating, sum of boxes, is 1.0. ADAS 11 and 13 scores are 4.67 and 4.67 respectively, with a score of 1.0 in delayed word recall... |\n",
|
||||
"| APOE4 | 1.0 |\n",
|
||||
"| TAU | 212.5 |\n",
|
||||
"| ... | ... |\n",
|
||||
"| MMSE | 29.0 | Volumetric Data: Under MRI conditions at a field strength of 1.5 Tesla MRI Tesla, using Cross Sectional FreeSurfer (FreeSurfer Version 4.3), the imaging data recorded includes ventricles volume at 54422.0, hippocampus volume at 6677.0, whole brain volume at 1147980.0, entorhinal cortex volume at 2782.0, fusiform gyrus volume at 19432.0, and middle temporal area volume at 24951.0. The intracranial volume measured is 1799580.0.... |\n",
|
||||
"| CDRSB | 0.0 |\n",
|
||||
"| ... | ... |\n",
|
||||
"| FLDSTRENG | 1.5 Tesla MRI |\n",
|
||||
"| Ventricles | 84599 |\n",
|
||||
"| Hippocampus | 5319 |\n",
|
||||
"| ... | ... |\n",
|
||||
"\n",
|
||||
"Figure 2: An example of a patient table and its corresponding clinical description.\n",
|
||||
"\n",
|
||||
"skills. Mathematics, as a highly structured and logic-driven discipline, provides an ideal testing ground for evaluating this reasoning ability. To investigate o1-preview's performance, we designed a series of tests covering various difficulty levels. We begin with high school-level math competition problems in this section, followed by college-level mathematics problems in the next section, allowing us to observe the model's logical reasoning across varying levels of complexity.\n",
|
||||
"\n",
|
||||
"In this section, we selected two primary areas of mathematics: algebra and counting and probability in this section. We chose these two topics because of their heavy reliance on problem-solving skills and their frequent use in assessing logical and abstract thinking [46]. The dataset used in testing is from the MATH dataset [46]. The problems in the dataset cover a wide range of subjects, including Prealgebra, Intermediate Algebra, Algebra, Geometry, Counting and Probability, Number Theory, and Precalculus. Each problem is categorized based on difficulty, ranked from level 1 to 5, according to the Art of Problem Solving (AoPS). The dataset mainly comprises problems from various high school math competitions, including the American Mathematics Competitions (AMC) 10 and 12, as well as the American Invitational Mathematics Examination (AIME), and other similar contests. Each problem comes with detailed reference solutions, allowing for a comprehensive comparison of o1-preview's solutions.\n",
|
||||
"\n",
|
||||
"In addition to evaluating the final answers produced by o1-preview, our analysis delves into the step-by-step reasoning process of the o1-preview's solutions. By comparing o1-preview's solutions with the dataset's solutions, we assess its ability to engage in logical reasoning, handle abstract problem-solving tasks, and apply structured approaches to reach correct answers. This deeper analysis offers insights into o1-preview's overall reasoning capabilities, using mathematics as a reliable indicator for logical and structured thought processes.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# using Sonnet-3.5\n",
|
||||
"print(docs[0].get_content(metadata_mode=\"all\"))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"id": "1511a30f-3efc-4142-9668-7dc056a24d0c",
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "1511a30f-3efc-4142-9668-7dc056a24d0c",
|
||||
"outputId": "2e5e8e20-2b41-4183-f21f-dff503a03089"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"page: 25\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"| Participant_ID | clinical Description Reference |\n",
|
||||
"|----------------|--------------------------------|\n",
|
||||
"| **Attribute** | **Value** |\n",
|
||||
"| Age | 72.0 |\n",
|
||||
"| Sex | Female |\n",
|
||||
"| Education | 15 |\n",
|
||||
"| Race | White |\n",
|
||||
"| DX_bl | AD |\n",
|
||||
"| DX | Dementia |\n",
|
||||
"| ... | ... |\n",
|
||||
"| APOE4 | 1.0 |\n",
|
||||
"| TAU | 212.5 |\n",
|
||||
"| ... | ... |\n",
|
||||
"| MMSE | 29.0 |\n",
|
||||
"| CDRSB | 0.0 |\n",
|
||||
"| ... | ... |\n",
|
||||
"| FLDSTRENG | 1.5 Tesla MRI |\n",
|
||||
"| Ventricles | 84599 |\n",
|
||||
"| Hippocampus | 5319 |\n",
|
||||
"| ... | ... |\n",
|
||||
"\n",
|
||||
"**Basic Personal Information:** Subject 098_S_0896 is a 72.0-year-old Female who has completed 15 years of education. The ethnicity is Not Hisp/Latino and race is White. Marital status is Married. Initially diagnosed as AD, as of the date 2007-10-24, the final diagnosis was Dementia.\n",
|
||||
"\n",
|
||||
"**Biomarker Measurements:** The subject's genetic profile includes an ApoE4 status of 0.0...\n",
|
||||
"\n",
|
||||
"**Cognitive and Neurofunctional Assessments:** The Mini-Mental State Examination score stands at 29.0. The Clinical Dementia Rating, sum of boxes, is 1.0. ADAS 11 and 13 scores are 4.67 and 4.67 respectively, with a score of 1.0 in delayed word recall...\n",
|
||||
"\n",
|
||||
"**Volumetric Data:** Under MRI conditions at a field strength of 1.5 Tesla MRI Tesla, using Cross-Sectional FreeSurfer (FreeSurfer Version 4.3), the imaging data recorded includes ventricles volume at 84422.0, hippocampus volume at 6677.0, whole brain volume at 1147980.0, entorhinal cortex volume at 27820.0, fusiform gyrus volume at 19432.0, and middle temporal area volume at 24951.0. The intracranial volume measured is 1799580.0...\n",
|
||||
"\n",
|
||||
"Figure 2: An example of a patient table and its corresponding clinical description.\n",
|
||||
"\n",
|
||||
"----\n",
|
||||
"\n",
|
||||
"Skills. Mathematics, as a highly structured and logic-driven discipline, provides an ideal testing ground for evaluating this reasoning ability. To investigate o1-preview’s performance, we designed a series of tests covering various difficulty levels. We begin with high school-level math competition problems in this section, followed by college-level mathematics problems in the next section, allowing us to observe the model’s logical reasoning across varying levels of complexity.\n",
|
||||
"\n",
|
||||
"In this section, we selected two primary areas of mathematics: algebra and counting and probability in this section. We chose these two topics because of their heavy reliance on problem-solving skills and their frequent use in assessing logical and abstract thinking [46]. The dataset used in testing is from the MATH dataset [46]. The problems in the dataset cover a wide range of subjects, including Prealgebra, Intermediate Algebra, Algebra, Geometry, Counting and Probability, Number Theory, and Precalculus. Each problem is categorized based on difficulty, ranked from level 1 to 5, according to the Art of Problem Solving (AoPS). The dataset mainly comprises problems from various high school math competitions, including the American Mathematics Competitions (AMC) 10 and 12, as well as the American Invitational Mathematics Examination (AIME), and other similar contests. Each problem comes with detailed reference solutions, allowing for a comprehensive comparison of o1-preview’s solutions.\n",
|
||||
"\n",
|
||||
"In addition to evaluating the final answers produced by o1-preview, our analysis delves into the step-by-step reasoning process of the o1-preview’s solutions. By comparing o1-preview’s solutions with the dataset’s solutions, we assess its ability to engage in logical reasoning, handle abstract problem-solving tasks, and apply structured approaches to reach correct answers. This deeper analysis offers insights into o1-preview’s overall reasoning capabilities, using mathematics as a reliable indicator for logical and structured thought processes.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# using GPT-4o\n",
|
||||
"print(docs_gpt4o[0].get_content(metadata_mode=\"all\"))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "1c75bb85",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"provenance": []
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "llamacloud",
|
||||
"language": "python",
|
||||
"name": "llamacloud"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.12.4"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@@ -0,0 +1,181 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"<a href=\"https://colab.research.google.com/github/run-llama/llama_parse/blob/main/examples/demo_starter_parse_selected_pages.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Parse Selected Pages \n",
|
||||
"\n",
|
||||
"In this notebook we will demonstrate how to parse selected pages in a document using LlamaParse."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Installation\n",
|
||||
"\n",
|
||||
"Here we install `llama-parse` used for parsing the document"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!pip install llama-parse"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Set API Key"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# llama-parse is async-first, running the async code in a notebook requires the use of nest_asyncio\n",
|
||||
"import nest_asyncio\n",
|
||||
"\n",
|
||||
"nest_asyncio.apply()\n",
|
||||
"\n",
|
||||
"import os\n",
|
||||
"\n",
|
||||
"# API access to llama-cloud\n",
|
||||
"os.environ[\"LLAMA_CLOUD_API_KEY\"] = \"<YOUR LLAMACLOUD API KEY>\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Download Data\n",
|
||||
"\n",
|
||||
"Here we download Uber 2021 10K SEC filings data for the demonstration."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"--2024-12-05 11:40:59-- https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/10k/uber_2021.pdf\n",
|
||||
"Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 2606:50c0:8000::154, 2606:50c0:8002::154, 2606:50c0:8003::154, ...\n",
|
||||
"Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|2606:50c0:8000::154|:443... connected.\n",
|
||||
"HTTP request sent, awaiting response... 200 OK\n",
|
||||
"Length: 1880483 (1.8M) [application/octet-stream]\n",
|
||||
"Saving to: ‘./uber_2021.pdf’\n",
|
||||
"\n",
|
||||
"./uber_2021.pdf 100%[===================>] 1.79M --.-KB/s in 0.1s \n",
|
||||
"\n",
|
||||
"2024-12-05 11:40:59 (14.2 MB/s) - ‘./uber_2021.pdf’ saved [1880483/1880483]\n",
|
||||
"\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/10k/uber_2021.pdf' -O './uber_2021.pdf'"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Parse the PDF file in selected pages\n",
|
||||
"\n",
|
||||
"Here we will parse the PDF file in selected pages and get the text in `markdown` format."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Started parsing the file under job_id ad1087c1-b085-4dc7-9aa8-d13cdd440f2b\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from llama_parse import LlamaParse\n",
|
||||
"\n",
|
||||
"parser = LlamaParse(\n",
|
||||
" target_pages=\"0,1,2\",\n",
|
||||
" result_type=\"markdown\"\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"documents = parser.load_data('./uber_2021.pdf')"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[Document(id_='d0b34f4a-27ef-48e2-a92a-386e5e265f4c', embedding=None, metadata={}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={}, metadata_template='{key}: {value}', metadata_separator='\\n', text='# UNITED STATES SECURITIES AND EXCHANGE COMMISSION\\n\\n# Washington, D.C. 20549\\n\\n# FORM 10-K\\n\\n(Mark One)\\n\\n☒ ANNUAL REPORT PURSUANT TO SECTION 13 OR 15(d) OF THE SECURITIES EXCHANGE ACT OF 1934\\n\\nFor the fiscal year ended December 31, 2021\\n\\nOR\\n\\n☐ TRANSITION REPORT PURSUANT TO SECTION 13 OR 15(d) OF THE SECURITIES EXCHANGE ACT OF 1934\\n\\nFor the transition period from _____ to _____\\n\\nCommission File Number: 001-38902\\n\\n# UBER TECHNOLOGIES, INC.\\n\\n(Exact name of registrant as specified in its charter)\\n\\nDelaware\\n\\n45-2647441\\n\\n(State or other jurisdiction of incorporation or organization) (I.R.S. Employer Identification No.)\\n\\n1515 3rd Street\\n\\nSan Francisco, California 94158\\n\\n(Address of principal executive offices, including zip code)\\n\\n(415) 612-8582\\n\\n(Registrant’s telephone number, including area code)\\n\\n# Securities registered pursuant to Section 12(b) of the Act:\\n\\n|Title of each class|Trading Symbol(s)|Name of each exchange on which registered|\\n|---|---|---|\\n|Common Stock, par value $0.00001 per share|UBER|New York Stock Exchange|\\n\\nSecurities registered pursuant to Section 12(g) of the Act: None\\n\\nIndicate by check mark whether the registrant is a well-known seasoned issuer, as defined in Rule 405 of the Securities Act. Yes ☒ No ☐\\n\\nIndicate by check mark whether the registrant is not required to file reports pursuant to Section 13 or Section 15(d) of the Act. Yes ☐ No ☒\\n\\nIndicate by check mark whether the registrant (1) has filed all reports required to be filed by Section 13 or 15(d) of the Securities Exchange Act of 1934 during the preceding 12 months (or for such shorter period that the registrant was required to file such reports), and (2) has been subject to such filing requirements for the past 90 days. Yes ☒ No ☐\\n\\nIndicate by check mark whether the registrant has submitted electronically every Interactive Data File required to be submitted pursuant to Rule 405 of Regulation S-T (§232.405 of this chapter) during the preceding 12 months (or for such shorter period that the registrant was required to submit such files). Yes ☒ No ☐\\n\\nIndicate by check mark whether the registrant is a large accelerated filer, an accelerated filer, a non-accelerated filer, a smaller reporting company, or an emerging growth company. See the definitions of “large accelerated filer,” “accelerated filer,” “smaller reporting company,” and “emerging growth company” in Rule 12b-2 of the Exchange Act.', mimetype='text/plain', start_char_idx=None, end_char_idx=None, metadata_seperator='\\n', text_template='{metadata_str}\\n\\n{content}'),\n",
|
||||
" Document(id_='253b1141-a260-466e-b164-b39df67ef799', embedding=None, metadata={}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={}, metadata_template='{key}: {value}', metadata_separator='\\n', text=\"# Large accelerated filer\\n\\n☒\\n\\n# Accelerated filer\\n\\n☐\\n\\n# Non-accelerated filer\\n\\n☐\\n\\n# Smaller reporting company\\n\\n☐\\n\\n# Emerging growth company\\n\\n☐\\n\\nIf an emerging growth company, indicate by check mark if the registrant has elected not to use the extended transition period for complying with any new or revised financial accounting standards provided pursuant to Section 13(a) of the Exchange Act.\\n\\n☐\\n\\nIndicate by check mark whether the registrant has filed a report on and attestation to its management’s assessment of the effectiveness of its internal control over financial reporting under Section 404(b) of the Sarbanes-Oxley Act (15 U.S.C. 7262(b)) by the registered public accounting firm that prepared or issued\\n\\n☒\\n\\nIndicate by check mark whether the registrant is a shell company (as defined in Rule 12b-2 of the Exchange Act). Yes\\n\\n☐\\n\\nNo\\n\\n☒\\n\\nThe aggregate market value of the voting and non-voting common equity held by non-affiliates of the registrant as of June 30, 2021, the last business day of the registrant's most recently completed second fiscal quarter, was approximately $90.5 billion based upon the closing price reported for such date on the New York Stock Exchange.\\n\\nThe number of shares of the registrant's common stock outstanding as of February 22, 2022 was 1,954,464,088.\\n\\n# DOCUMENTS INCORPORATED BY REFERENCE\\n\\nPortions of the registrant’s Definitive Proxy Statement relating to the Annual Meeting of Stockholders are incorporated by reference into Part III of this Annual Report on Form 10-K where indicated. Such Definitive Proxy Statement will be filed with the Securities and Exchange Commission within 120 days after the end of the registrant’s fiscal year ended December 31, 2021.\", mimetype='text/plain', start_char_idx=None, end_char_idx=None, metadata_seperator='\\n', text_template='{metadata_str}\\n\\n{content}'),\n",
|
||||
" Document(id_='ad988239-3ab5-498d-85ba-a29241db24d4', embedding=None, metadata={}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={}, metadata_template='{key}: {value}', metadata_separator='\\n', text='# UBER TECHNOLOGIES, INC.\\n\\n# TABLE OF CONTENTS\\n\\n|Special Note Regarding Forward-Looking Statements|2|\\n|---|---|\\n|PART I|PART I|\\n|Item 1. Business|4|\\n|Item 1A. Risk Factors|11|\\n|Item 1B. Unresolved Staff Comments|46|\\n|Item 2. Properties|46|\\n|Item 3. Legal Proceedings|46|\\n|Item 4. Mine Safety Disclosures|47|\\n|PART II|PART II|\\n|Item 5. Market for Registrant’s Common Equity, Related Stockholder Matters and Issuer Purchases of Equity Securities|47|\\n|Item 6. [Reserved]|48|\\n|Item 7. Management’s Discussion and Analysis of Financial Condition and Results of Operations|48|\\n|Item 7A. Quantitative and Qualitative Disclosures About Market Risk|69|\\n|Item 8. Financial Statements and Supplementary Data|70|\\n|Item 9. Changes in and Disagreements with Accountants on Accounting and Financial Disclosure|146|\\n|Item 9A. Controls and Procedures|147|\\n|Item 9B. Other Information|147|\\n|Item 9C. Disclosure Regarding Foreign Jurisdictions that Prevent Inspections|147|\\n|PART III|PART III|\\n|Item 10. Directors, Executive Officers and Corporate Governance|147|\\n|Item 11. Executive Compensation|147|\\n|Item 12. Security Ownership of Certain Beneficial Owners and Management and Related Stockholder Matters|148|\\n|Item 13. Certain Relationships and Related Transactions, and Director Independence|148|\\n|Item 14. Principal Accounting Fees and Services|148|\\n|PART IV|PART IV|\\n|Item 15. Exhibits, Financial Statement Schedules|148|\\n|Item 16. Form 10-K Summary|148|\\n|Exhibit Index|149|\\n|Signatures|152|', mimetype='text/plain', start_char_idx=None, end_char_idx=None, metadata_seperator='\\n', text_template='{metadata_str}\\n\\n{content}')]"
|
||||
]
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"documents"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "llamacloud",
|
||||
"language": "python",
|
||||
"name": "llamacloud"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.12.4"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
@@ -11,6 +11,8 @@
|
||||
"\n",
|
||||
"In this cookbook we show you how to build a multimodal report generation agent from a bank of research reports. We use the a set of ICLR papers (which were also used as the dataset in our [DeepLearning.ai course](https://www.deeplearning.ai/short-courses/building-agentic-rag-with-llamaindex/?utm_campaign=llamaindexC2-launch&utm_medium=headband&utm_source=dlai-homepage).\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"We use our workflow abstraction to define an agentic system that contains two main phases: a research phase that pulls in relevant files through chunk-level or file-level retrieval, and then a blog generation phase that synthesizes the final report."
|
||||
]
|
||||
},
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 1.5 MiB |
@@ -18,7 +18,9 @@
|
||||
"We index a set of relevant documents that Microsoft has - including its annual report, wikipedia page on Microsoft Azure, a slide deck on the government cloud and cybersecurity capabilities. We then help you build an agentic workflow that can ingest an RFP, and generate a response for it in \n",
|
||||
"a way that adheres to its guidelines.\n",
|
||||
"\n",
|
||||
"We use LlamaParse to parse the context documents as well as the RFP document itself."
|
||||
"We use LlamaParse to parse the context documents as well as the RFP document itself.\n",
|
||||
"\n",
|
||||
"**NOTE**: If you want to skip the indexing complexity and use LlamaCloud instead, check out the [RFP Example using LlamaCloud](https://github.com/run-llama/llamacloud-demo/blob/main/examples/report_generation/rfp_response/generate_rfp.ipynb)."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -26,7 +28,15 @@
|
||||
"execution_count": null,
|
||||
"id": "c4f28c6a-cb5e-4c16-bdc7-a69817fc4c12",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"The history saving thread hit an unexpected error (OperationalError('attempt to write a readonly database')).History will not be written to the database.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import nest_asyncio\n",
|
||||
"\n",
|
||||
@@ -408,6 +418,7 @@
|
||||
"from llama_index.core.llms import ChatMessage, MessageRole\n",
|
||||
"import logging\n",
|
||||
"import json\n",
|
||||
"import os\n",
|
||||
"\n",
|
||||
"_logger = logging.getLogger(__name__)\n",
|
||||
"_logger.setLevel(logging.INFO)\n",
|
||||
@@ -438,10 +449,22 @@
|
||||
"with our downstream research assistant, and the combined\n",
|
||||
"question:answer pairs will constitute the full RFP response.\n",
|
||||
"\n",
|
||||
"- Make sure the questions are comprehensive and adheres to the RFP requirements.\n",
|
||||
"You must TRY to extract out questions that can be answered by the provided knowledge base. We provide the list of file metadata below. \n",
|
||||
"\n",
|
||||
"Additional requirements:\n",
|
||||
"- Try to make the questions SPECIFIC given your knowledge of the RFP and the knowledge base. Instead of asking a question like \\\n",
|
||||
"\"How do we ensure security\" ask a question that actually addresses a security requirement in the RFP and can be addressed by the knowledge base.\n",
|
||||
"- Make sure the questions are comprehensive and addresses all the RFP requirements.\n",
|
||||
"- Make sure each question is descriptive - this gives our downstream assistant context to fill out the value for that question \n",
|
||||
"- Extract out all the questions as a list of strings.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Knowledge Base Files:\n",
|
||||
"{file_metadata}\n",
|
||||
"\n",
|
||||
"RFP Full Template:\n",
|
||||
"{rfp_text}\n",
|
||||
"\n",
|
||||
"\"\"\"\n",
|
||||
"\n",
|
||||
"# this is the prompt that generates the final RFP response given the original template text and question-answer pairs.\n",
|
||||
@@ -513,6 +536,7 @@
|
||||
" output_dir: str = data_out_dir,\n",
|
||||
" agent_system_prompt: str = AGENT_SYSTEM_PROMPT,\n",
|
||||
" generate_output_prompt: str = GENERATE_OUTPUT_PROMPT,\n",
|
||||
" extract_keys_prompt: str = EXTRACT_KEYS_PROMPT,\n",
|
||||
" **kwargs,\n",
|
||||
" ) -> None:\n",
|
||||
" \"\"\"Init params.\"\"\"\n",
|
||||
@@ -527,11 +551,13 @@
|
||||
" self.output_dir = output_dir\n",
|
||||
"\n",
|
||||
" self.agent_system_prompt = agent_system_prompt\n",
|
||||
" self.extract_keys_prompt = extract_keys_prompt\n",
|
||||
"\n",
|
||||
" # if not exists, create\n",
|
||||
" out_path = Path(self.output_dir) / \"workflow_output\"\n",
|
||||
" if not out_path.exists():\n",
|
||||
" out_path.mkdir(parents=True, exist_ok=True)\n",
|
||||
" os.chmod(str(out_path), 0o0777)\n",
|
||||
"\n",
|
||||
" self.generate_output_prompt = PromptTemplate(generate_output_prompt)\n",
|
||||
"\n",
|
||||
@@ -571,12 +597,31 @@
|
||||
" else:\n",
|
||||
" # try stuffing all text into the prompt\n",
|
||||
" all_text = \"\\n\\n\".join([d.get_content(metadata_mode=\"all\") for d in docs])\n",
|
||||
" prompt = PromptTemplate(template=EXTRACT_KEYS_PROMPT)\n",
|
||||
" prompt = PromptTemplate(template=self.extract_keys_prompt)\n",
|
||||
"\n",
|
||||
" file_metadata = \"\\n\\n\".join(\n",
|
||||
" [\n",
|
||||
" f\"Name:{t.metadata.name}\\nDescription:{t.metadata.description}\"\n",
|
||||
" for t in tools\n",
|
||||
" ]\n",
|
||||
" )\n",
|
||||
" try:\n",
|
||||
" if self._verbose:\n",
|
||||
" ctx.write_event_to_stream(\n",
|
||||
" LogEvent(msg=\">> Extracting questions from LLM\")\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" output_qs = self.llm.structured_predict(\n",
|
||||
" OutputQuestions, prompt, context=all_text\n",
|
||||
" OutputQuestions,\n",
|
||||
" prompt,\n",
|
||||
" file_metadata=file_metadata,\n",
|
||||
" rfp_text=all_text,\n",
|
||||
" ).questions\n",
|
||||
"\n",
|
||||
" if self._verbose:\n",
|
||||
" qs_text = \"\\n\".join([f\"* {q}\" for q in output_qs])\n",
|
||||
" ctx.write_event_to_stream(LogEvent(msg=f\">> Questions:\\n{qs_text}\"))\n",
|
||||
"\n",
|
||||
" except Exception as e:\n",
|
||||
" _logger.error(f\"Error extracting questions from page: {all_text}\")\n",
|
||||
" _logger.error(e)\n",
|
||||
@@ -730,158 +775,222 @@
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Running step parse_output_template\n",
|
||||
"Started parsing the file under job_id ad74a9de-c9d2-44b8-ad09-8a7d8baf383f\n",
|
||||
"Step parse_output_template produced event OutputTemplateEvent\n",
|
||||
"Running step extract_questions\n",
|
||||
"Step extract_questions produced no event\n",
|
||||
"Running step handle_question\n",
|
||||
">> Extracting questions from LLM\n",
|
||||
">> Questions:\n",
|
||||
"* What are the specific security requirements for the JEDI Cloud as outlined in the RFP, and how can they be addressed using Microsoft Azure Government's compliance standards and security features?\n",
|
||||
"* How does the RFP define the requirements for high availability and failover in cloud services, and what solutions does Microsoft Azure offer to meet these requirements?\n",
|
||||
"* What are the RFP's stipulations regarding data portability and interoperability, and how can Microsoft Azure's services facilitate these processes?\n",
|
||||
"* What are the expectations for program management and oversight as per the RFP, and how can Microsoft Azure's management tools and practices align with these expectations?\n",
|
||||
"* How does the RFP address the need for logical isolation and secure data transfer, and what capabilities does Microsoft Azure provide to ensure these requirements are met?\n",
|
||||
"* What are the RFP's requirements for tactical edge capabilities, and how can Microsoft Azure's solutions support operations in communication-degraded or disconnected environments?\n",
|
||||
"* What does the RFP specify about the integration of third-party services and applications, and how can Microsoft Azure's marketplace and APIs support this integration?\n",
|
||||
"* How does the RFP outline the approach to small business participation, and what strategies can be employed to maximize small business involvement using Microsoft Azure's ecosystem?\n",
|
||||
"* What are the RFP's criteria for evaluating cloud service automation, and how can Microsoft Azure's automation tools and APIs fulfill these criteria?\n",
|
||||
"* What are the RFP's requirements for compliance with federal regulations and standards, and how does Microsoft Azure ensure adherence to these regulations?\n",
|
||||
"Running step handle_question\n",
|
||||
"Running step handle_question\n",
|
||||
"Running step handle_question\n",
|
||||
"Step handle_question produced event QuestionAnsweredEvent\n",
|
||||
"Running step handle_question\n",
|
||||
">> Asked question: What are the specific deliverables required for this project?\n",
|
||||
">> Got response: To determine the specific deliverables required for the project, I need more context about the project itself. Could you please provide more details or specify the project you are referring to?\n",
|
||||
">> Asked question: How does the RFP define the requirements for high availability and failover in cloud services, and what solutions does Microsoft Azure offer to meet these requirements?\n",
|
||||
">> Got response: The RFP (Request for Proposal) does not explicitly define the requirements for high availability and failover in the provided context from the Azure Government document. However, Microsoft Azure offers several solutions to meet high availability and failover requirements:\n",
|
||||
"\n",
|
||||
"1. **Azure Site Recovery**: This service provides a comprehensive disaster recovery solution, allowing for site-to-Azure, any cloud, and site-to-site recovery options. It supports various operating systems, including Windows and Linux, and facilitates migration from AWS, VMware, Hyper-V, and physical servers to Azure Cloud.\n",
|
||||
"\n",
|
||||
"2. **Azure Kubernetes Service (AKS)**: AKS enables the deployment of production-ready Kubernetes clusters in Azure, which can be used to ensure high availability and manage failover scenarios.\n",
|
||||
"\n",
|
||||
"3. **Azure Functions**: These are used in serverless computing architectures, allowing for event-driven execution without managing server resources, which can contribute to high availability by automatically scaling based on demand.\n",
|
||||
"\n",
|
||||
"4. **Azure Stack HCI**: This hyper-converged infrastructure product allows for running virtualized workloads on-premises while connecting to Azure for cloud services, providing a hybrid solution for high availability.\n",
|
||||
"\n",
|
||||
"5. **Azure Fabric Controller**: This component maintains the scalability and dependability of services and environments in the data center, preventing failures and managing web applications, memory allocation, and load balancing.\n",
|
||||
"\n",
|
||||
"These solutions collectively help ensure that Microsoft Azure can meet high availability and failover requirements for cloud services.\n",
|
||||
"Running step combine_answers\n",
|
||||
"Step combine_answers produced no event\n",
|
||||
"Step handle_question produced event QuestionAnsweredEvent\n",
|
||||
"Running step handle_question\n",
|
||||
">> Asked question: What is the overall project scope and objectives as outlined in the RFP?\n",
|
||||
">> Got response: To answer this question, I would need to access the specific RFP document related to the project you are referring to. Could you please provide more details or specify the document you are referring to?\n",
|
||||
">> Asked question: What are the specific security requirements for the JEDI Cloud as outlined in the RFP, and how can they be addressed using Microsoft Azure Government's compliance standards and security features?\n",
|
||||
">> Got response: The JEDI Cloud security requirements as outlined in the RFP are not directly available in the retrieved documents. However, Microsoft Azure Government provides a robust set of compliance standards and security features that can address typical security requirements for government cloud services, which may align with JEDI Cloud's needs. Here are some key points:\n",
|
||||
"\n",
|
||||
"1. **Compliance Standards**: Microsoft Azure Government is compliant with a wide range of certifications and standards, including:\n",
|
||||
" - FedRAMP Moderate and High JAB P-ATO\n",
|
||||
" - DoD DISA SRG Levels 2, 4, and 5\n",
|
||||
" - NIST SP 800-171\n",
|
||||
" - FIPS 140-2\n",
|
||||
" - CJIS (Criminal Justice Information Services)\n",
|
||||
" - ITAR (International Traffic in Arms Regulations)\n",
|
||||
" - IRS 1075\n",
|
||||
"\n",
|
||||
"2. **Security Features**:\n",
|
||||
" - **Dedicated Physical Network**: Azure Government provides a dedicated physical instance of Microsoft Azure with a dedicated network, offering geo-replication between locations.\n",
|
||||
" - **ExpressRoute**: Dedicated ExpressRoute sites for government are included in Azure Government's DoD and FedRAMP compliance scope, ensuring secure and reliable connectivity.\n",
|
||||
" - **Identity and Access Management**: Azure Active Directory provides consistent identity management across cloud and on-premises environments.\n",
|
||||
" - **Integrated Management and Security**: Azure offers integrated management and security features to ensure consistent data platform and unified development across environments.\n",
|
||||
"\n",
|
||||
"3. **CJIS Compliance**: Microsoft has committed to CJIS regulations in multiple states, allowing law enforcement agencies to leverage cloud-based solutions for criminal justice applications.\n",
|
||||
"\n",
|
||||
"These compliance standards and security features make Microsoft Azure Government a suitable platform to meet the stringent security requirements typically associated with government cloud services like JEDI Cloud.\n",
|
||||
"Running step combine_answers\n",
|
||||
"Step combine_answers produced no event\n",
|
||||
"Step handle_question produced event QuestionAnsweredEvent\n",
|
||||
"Running step handle_question\n",
|
||||
">> Asked question: What is the budget range for the project, and what are the payment terms?\n",
|
||||
">> Got response: To answer your question about the budget range for the project and the payment terms, I would need more specific context or a document related to the project you are referring to. Could you please provide more details or specify the document or context in which this project is mentioned?\n",
|
||||
">> Asked question: What are the RFP's stipulations regarding data portability and interoperability, and how can Microsoft Azure's services facilitate these processes?\n",
|
||||
">> Got response: The RFP's stipulations regarding data portability and interoperability are not explicitly detailed in the retrieved documents from the Azure Government file. However, Microsoft Azure's services facilitate data portability and interoperability through various offerings:\n",
|
||||
"\n",
|
||||
"1. **Data Management and Integration Services**: Azure provides a range of data management services such as Azure Data Explorer for big data analytics, Azure Data Factory for data integration and workflow automation, and Azure Synapse Analytics for cloud data warehousing. These services enable seamless data movement and transformation across different platforms and environments.\n",
|
||||
"\n",
|
||||
"2. **Hybrid Cloud Architecture**: Azure supports a hybrid cloud architecture that combines cloud and on-premises components. This architecture ensures consistent identity, integrated management, and a consistent data platform, which are crucial for interoperability between different systems.\n",
|
||||
"\n",
|
||||
"3. **Storage Services**: Azure offers various storage services like Azure Blob Storage for unstructured data, Azure Table Storage for structured data, and Azure Queue Storage for asynchronous messaging. These services provide REST and SDK APIs for accessing and managing data, facilitating data portability across different applications and platforms.\n",
|
||||
"\n",
|
||||
"4. **Communication and Messaging Services**: Azure Service Bus and Event Hubs support scalable and reliable communication mechanisms, enabling interoperability in service-oriented architectures.\n",
|
||||
"\n",
|
||||
"5. **Open and Hybrid Solutions**: Azure's commitment to open and hybrid solutions allows for integration with existing on-premises systems and other cloud services, enhancing interoperability.\n",
|
||||
"\n",
|
||||
"These services collectively support the goals of data portability and interoperability by providing flexible, scalable, and integrated solutions that can work across various environments and platforms.\n",
|
||||
"Running step combine_answers\n",
|
||||
"Step combine_answers produced no event\n",
|
||||
"Step handle_question produced event QuestionAnsweredEvent\n",
|
||||
"Running step handle_question\n",
|
||||
">> Asked question: What is the timeline for the project, including key milestones and deadlines?\n",
|
||||
">> Got response: The retrieved documents do not provide a specific timeline for a project, including key milestones and deadlines. They contain various timelines related to Microsoft's history, product launches, and strategic initiatives, but none specifically outline a project timeline with milestones and deadlines.\n",
|
||||
">> Asked question: What are the expectations for program management and oversight as per the RFP, and how can Microsoft Azure's management tools and practices align with these expectations?\n",
|
||||
">> Got response: To address the expectations for program management and oversight as per the RFP and how Microsoft Azure's management tools and practices align with these expectations, let's break down the information retrieved:\n",
|
||||
"\n",
|
||||
"If you have a specific project in mind, please provide more details so I can assist you better. Alternatively, if you are looking for general information on how Microsoft or Azure typically handles project timelines, I can help with that as well.\n",
|
||||
"### Expectations for Program Management and Oversight (RFP Context)\n",
|
||||
"The retrieved documents from the Azure Government file did not provide specific details about the expectations for program management and oversight as per an RFP. However, typically, RFPs for government projects emphasize the need for:\n",
|
||||
"- **Compliance with Standards**: Ensuring adherence to relevant compliance and security standards.\n",
|
||||
"- **Robust Security Measures**: Implementing strong security protocols to protect sensitive data.\n",
|
||||
"- **Efficient Resource Management**: Effective management of resources to ensure project timelines and budgets are met.\n",
|
||||
"- **Regular Reporting and Monitoring**: Continuous monitoring and reporting to track progress and address issues promptly.\n",
|
||||
"- **Risk Management**: Identifying and mitigating potential risks throughout the project lifecycle.\n",
|
||||
"\n",
|
||||
"### Microsoft Azure's Management Tools and Practices\n",
|
||||
"From the Azure Wiki file, we can see that Microsoft Azure offers several management tools and practices that align with typical RFP expectations:\n",
|
||||
"- **Azure Resource Manager**: Allows users to group related services, making it easier to deploy, manage, and monitor resources efficiently.\n",
|
||||
"- **Azure Portal**: A web-based interface for managing Azure services, providing capabilities to browse active resources, adjust settings, and view monitoring data.\n",
|
||||
"- **Compliance and Security**: Azure complies with numerous global, US government, industry, and regional standards, ensuring robust security and compliance.\n",
|
||||
"- **Azure Functions and IoT Services**: These services support event-driven architectures and IoT management, which can be crucial for projects requiring real-time data processing and management.\n",
|
||||
"- **Azure Fabric Controller**: Manages scalability and dependability of services, preventing failures and ensuring efficient resource allocation.\n",
|
||||
"\n",
|
||||
"### Alignment with RFP Expectations\n",
|
||||
"Microsoft Azure's management tools and practices align well with typical RFP expectations through:\n",
|
||||
"- **Comprehensive Compliance**: Azure's adherence to various compliance standards ensures that projects meet regulatory requirements.\n",
|
||||
"- **Efficient Management**: Tools like Azure Resource Manager and Azure Portal facilitate efficient resource management and monitoring.\n",
|
||||
"- **Security and Risk Management**: Azure's security measures and compliance with standards like FedRAMP and NIST SP 800-171 help in managing risks effectively.\n",
|
||||
"- **Scalability and Flexibility**: Azure's infrastructure supports scalable and flexible deployment models, which are essential for adapting to project needs.\n",
|
||||
"\n",
|
||||
"In summary, while specific RFP expectations were not detailed in the retrieved documents, Microsoft Azure's comprehensive suite of management tools and compliance with security standards make it well-suited to meet typical program management and oversight requirements in government projects.\n",
|
||||
"Running step combine_answers\n",
|
||||
"Step combine_answers produced no event\n",
|
||||
"Step handle_question produced event QuestionAnsweredEvent\n",
|
||||
"Running step handle_question\n",
|
||||
">> Asked question: What are the evaluation criteria that will be used to assess proposals?\n",
|
||||
">> Got response: The retrieved documents did not provide specific information on the evaluation criteria used to assess proposals. Based on my training data, evaluation criteria for proposals typically include factors such as:\n",
|
||||
">> Asked question: How does the RFP address the need for logical isolation and secure data transfer, and what capabilities does Microsoft Azure provide to ensure these requirements are met?\n",
|
||||
">> Got response: The RFP addresses the need for logical isolation and secure data transfer by leveraging Microsoft Azure Government's capabilities. Azure Government provides a physically isolated instance of Microsoft Azure, ensuring that data sovereignty remains within the U.S. This isolation is crucial for U.S. government entities that require secure and compliant cloud services. Additionally, Azure Government is operated by screened U.S. persons, further enhancing security and compliance.\n",
|
||||
"\n",
|
||||
"1. **Technical Merit**: The feasibility, innovation, and technical approach of the proposal.\n",
|
||||
"2. **Cost-Effectiveness**: The budget and financial plan, ensuring it is reasonable and justified.\n",
|
||||
"3. **Experience and Qualifications**: The expertise and past performance of the proposing team or organization.\n",
|
||||
"4. **Compliance and Risk Management**: Adherence to regulatory requirements and the ability to manage potential risks.\n",
|
||||
"5. **Impact and Benefits**: The potential positive outcomes and benefits of the proposal for the intended stakeholders.\n",
|
||||
"Microsoft Azure provides several capabilities to ensure logical isolation and secure data transfer:\n",
|
||||
"\n",
|
||||
"For specific criteria related to a particular organization or context, it would be best to refer to the official request for proposals (RFP) document or guidelines provided by the issuing entity.\n",
|
||||
"1. **Physical and Logical Isolation**: Azure Government offers a physically isolated cloud environment specifically for U.S. government entities, ensuring that data and operations are separate from the commercial Azure cloud.\n",
|
||||
"\n",
|
||||
"2. **Compliance and Security Standards**: Azure complies with numerous global, U.S. government, industry, and regional certifications and compliance standards, such as FedRAMP, DoD DISA SRG, and CJIS, which are critical for secure data handling and transfer.\n",
|
||||
"\n",
|
||||
"3. **Identity and Access Management**: Azure Active Directory and related services provide consistent identity management and secure access control across cloud and on-premises environments.\n",
|
||||
"\n",
|
||||
"4. **Data Protection**: Azure offers services like Azure Information Protection to safeguard sensitive information and ensure secure data transfer.\n",
|
||||
"\n",
|
||||
"5. **Blockchain and IoT Security**: Azure supports secure infrastructure for blockchain networks and IoT devices, ensuring secure data transfer and management.\n",
|
||||
"\n",
|
||||
"These capabilities collectively ensure that Microsoft Azure can meet the requirements for logical isolation and secure data transfer as outlined in the RFP.\n",
|
||||
"Running step combine_answers\n",
|
||||
"Step combine_answers produced no event\n",
|
||||
"Step handle_question produced event QuestionAnsweredEvent\n",
|
||||
"Running step handle_question\n",
|
||||
">> Asked question: What are the qualifications and experience required for the project team members?\n",
|
||||
">> Got response: The qualifications and experience required for project team members in the context of Microsoft and Azure can vary depending on the specific roles and responsibilities within a project. Here are some insights based on the retrieved documents:\n",
|
||||
">> Asked question: What are the RFP's requirements for tactical edge capabilities, and how can Microsoft Azure's solutions support operations in communication-degraded or disconnected environments?\n",
|
||||
">> Got response: ### RFP Requirements for Tactical Edge Capabilities\n",
|
||||
"\n",
|
||||
"1. **Azure Certifications and Roles**:\n",
|
||||
" - Microsoft offers a variety of certifications for different roles, such as Azure Data Scientist Associate, Azure Database Administrator Associate, Azure Developer Associate, Azure Security Engineer Associate, and more. These certifications indicate a level of expertise in specific areas of Azure services and solutions.\n",
|
||||
"The retrieved documents did not provide specific details about RFP requirements for tactical edge capabilities. However, Microsoft Azure Government is designed to meet various compliance and security standards, which could be relevant for tactical edge operations. Azure Government provides dedicated physical networks, geo-replication, and compliance with standards like FedRAMP and CJIS, which are critical for secure and reliable operations in government and defense sectors.\n",
|
||||
"\n",
|
||||
"2. **Cybersecurity Skills**:\n",
|
||||
" - There is a significant demand for cybersecurity professionals, and Microsoft has initiatives to address the skills gap. Training and certification in cybersecurity are crucial, and Microsoft collaborates with various organizations to provide training and certification opportunities.\n",
|
||||
"### Microsoft Azure's Solutions for Communication-Degraded or Disconnected Environments\n",
|
||||
"\n",
|
||||
"3. **Diversity and Inclusion**:\n",
|
||||
" - Microsoft emphasizes the importance of diversity and inclusion in its workforce. The company aims to recruit and retain talent from diverse backgrounds and experiences, which is crucial for fostering innovation and success.\n",
|
||||
"Microsoft Azure offers several solutions that can support operations in communication-degraded or disconnected environments:\n",
|
||||
"\n",
|
||||
"4. **Technical and Industry Skills**:\n",
|
||||
" - For roles related to security and threat intelligence, deep technical and industry skills are required. Teams like the Security Service Line (SSL) and Microsoft Security Response Center (MSRC) provide specialized services and require expertise in incident response, threat intelligence, and cyber resilience.\n",
|
||||
"1. **Azure IoT Edge**: This service allows cloud intelligence to be deployed locally on IoT edge devices. It enables the processing of data at the edge, reducing the need for constant cloud connectivity.\n",
|
||||
"\n",
|
||||
"5. **AI and Quantum Computing**:\n",
|
||||
" - With the rise of AI and quantum computing, skills in these areas are becoming increasingly important. Microsoft offers initiatives and training programs to develop AI skills, which are essential for leveraging AI technologies effectively.\n",
|
||||
"2. **Azure Orbital**: This service provides connectivity to remote locations without ground infrastructure by using satellite data. It can be particularly useful in areas where traditional communication networks are unavailable or unreliable.\n",
|
||||
"\n",
|
||||
"6. **Project Management and Collaboration**:\n",
|
||||
" - For roles involving project management and collaboration, skills in managing and coordinating projects, as well as working with diverse teams, are important. Microsoft Teams and other collaboration tools are used to facilitate communication and project execution.\n",
|
||||
"3. **Azure Service Bus**: Supports communication between applications running on Azure and off-premises devices, which can help maintain operations even when connectivity is intermittent.\n",
|
||||
"\n",
|
||||
"Overall, the qualifications and experience required for project team members at Microsoft and Azure involve a combination of technical expertise, certifications, diversity and inclusion awareness, and skills in emerging technologies like AI and cybersecurity.\n",
|
||||
"4. **Azure Stack HCI**: This infrastructure product allows for the running of virtualized workloads on-premises, which can be connected to Azure for cloud services, providing flexibility in disconnected environments.\n",
|
||||
"\n",
|
||||
"These solutions enable Azure to support operations in environments where communication may be degraded or disconnected, ensuring continuity and resilience in critical operations.\n",
|
||||
"Running step combine_answers\n",
|
||||
"Step combine_answers produced no event\n",
|
||||
"Step handle_question produced event QuestionAnsweredEvent\n",
|
||||
">> Asked question: What are the specific requirements for project management and reporting?\n",
|
||||
">> Got response: The specific requirements for project management and reporting, particularly in the context of Microsoft and its cloud services, can be derived from various compliance and operational standards. Here are some key points:\n",
|
||||
">> Asked question: What does the RFP specify about the integration of third-party services and applications, and how can Microsoft Azure's marketplace and APIs support this integration?\n",
|
||||
">> Got response: The integration of third-party services and applications in Microsoft Azure is supported through various features and services. Azure provides a platform that allows developers to build and deploy applications using multiple programming languages such as ASP.NET, PHP, Node.js, Java, or Python. These applications can be deployed using various methods like FTP, Git, Mercurial, Team Foundation Server, or through the user portal. Azure also offers a gallery of open-source applications that can be deployed, which is part of its platform as a service (PaaS) offerings.\n",
|
||||
"\n",
|
||||
"1. **Compliance and Security**: Microsoft has committed to various compliance standards such as CJIS (Criminal Justice Information Services) for law enforcement agencies, which includes requirements for personnel security, security training, and adherence to security policies. This is crucial for project management in sectors dealing with sensitive information.\n",
|
||||
"Azure's marketplace and APIs further support this integration by providing a wide range of services and tools. Azure offers REST and SDK APIs for storing and accessing data on the cloud, which facilitates the integration of third-party services. The Azure Marketplace is a platform where developers can find, try, and purchase applications and services that run on Azure. It provides a variety of solutions that can be integrated into existing applications, enhancing their functionality and performance.\n",
|
||||
"\n",
|
||||
"2. **Operational Risks**: Microsoft emphasizes the importance of maintaining a robust operations infrastructure to handle user traffic, service growth, and product complexity. This includes ensuring adequate data center capacity, internet connectivity, and power supply, which are critical for project management and reporting in cloud services.\n",
|
||||
"Additionally, Azure provides services like Azure Kubernetes Service (AKS) for deploying production-ready Kubernetes clusters, Azure Functions for serverless computing, and Azure IoT Hub for managing IoT assets. These services enable seamless integration and management of third-party applications and services within the Azure ecosystem.\n",
|
||||
"\n",
|
||||
"3. **Regulatory Requirements**: Microsoft is subject to a wide range of legal and regulatory requirements globally, including those related to data privacy, cybersecurity, and telecommunications. These regulations impact how projects are managed and reported, especially in terms of data handling and compliance.\n",
|
||||
"\n",
|
||||
"4. **Metrics and Reporting**: Microsoft uses various metrics to assess business performance and make informed decisions. These metrics are disclosed to provide transparency and reflect the evolution of products and services. This is part of the project management and reporting process to ensure alignment with business goals.\n",
|
||||
"\n",
|
||||
"5. **Cybersecurity and Infrastructure Resilience**: There are ongoing efforts to improve cybersecurity and infrastructure resilience, which include regulatory initiatives and standards for IoT and OT device security. These initiatives impact project management by setting requirements for security practices and reporting.\n",
|
||||
"\n",
|
||||
"6. **Supply Chain and Development Infrastructure**: Microsoft has developed tools and practices to secure its development infrastructure, including threat modeling and adopting secure boot for build machines. These practices are part of project management to ensure secure and efficient software development and deployment.\n",
|
||||
"\n",
|
||||
"These points highlight the multifaceted nature of project management and reporting requirements, which encompass compliance, operational efficiency, regulatory adherence, and security measures.\n",
|
||||
"Overall, Microsoft Azure's marketplace and APIs offer robust support for integrating third-party services and applications, providing developers with the tools and resources needed to build, deploy, and manage applications efficiently.\n",
|
||||
"Running step combine_answers\n",
|
||||
"Step combine_answers produced no event\n",
|
||||
"Step handle_question produced event QuestionAnsweredEvent\n",
|
||||
">> Asked question: What are the expectations for communication and collaboration with the client?\n",
|
||||
">> Got response: The expectations for communication and collaboration with clients, particularly in the context of Microsoft and its cloud services, can be summarized as follows:\n",
|
||||
">> Asked question: How does the RFP outline the approach to small business participation, and what strategies can be employed to maximize small business involvement using Microsoft Azure's ecosystem?\n",
|
||||
">> Got response: The retrieved documents did not provide specific details on how the RFP outlines the approach to small business participation. However, I can provide some general strategies that can be employed to maximize small business involvement using Microsoft Azure's ecosystem based on the information available:\n",
|
||||
"\n",
|
||||
"1. **Collaboration and Compliance**: Microsoft emphasizes the importance of collaboration with clients to ensure compliance with regulatory requirements. For example, in the context of the Criminal Justice Information Services (CJIS) Security Policy, Microsoft collaborates with law enforcement agencies to meet compliance requirements, such as background checks and security training (azure_gov.pdf).\n",
|
||||
"1. **Utilize Azure's Platform Services**: Small businesses can leverage Azure's Platform as a Service (PaaS) offerings, such as Azure Web Apps and Azure Functions, to build and deploy applications without the need to manage underlying infrastructure. This reduces costs and complexity, allowing small businesses to focus on innovation and growth.\n",
|
||||
"\n",
|
||||
"2. **Communication Services**: Microsoft Azure offers communication services that enable the creation of web and mobile communication applications, including SMS, video calling, and web-based chat. These services facilitate effective communication and collaboration with clients (azure_wiki.pdf).\n",
|
||||
"2. **Leverage Azure IoT and AI Capabilities**: Small businesses can take advantage of Azure IoT Hub and Azure Machine Learning to develop smart solutions that can enhance their products and services. These tools can help small businesses create innovative solutions in areas like smart devices, predictive maintenance, and data analytics.\n",
|
||||
"\n",
|
||||
"3. **Productivity and Collaboration Tools**: Microsoft provides a suite of productivity and collaboration tools, such as Microsoft 365, Dynamics 365, and Microsoft Teams, which are designed to enhance communication and collaboration within organizations. These tools help empower employees, optimize operations, and engage customers (msft_10k_2024.pdf).\n",
|
||||
"3. **Participate in Azure's Partner Network**: By joining the Microsoft Partner Network, small businesses can gain access to resources, training, and support to help them build and market their solutions. This network also provides opportunities for collaboration with other businesses and access to a broader customer base.\n",
|
||||
"\n",
|
||||
"4. **Collective Defense and Cybersecurity**: Microsoft emphasizes the importance of collaboration in cybersecurity, working with partners across the industry to enhance collective defense efforts. This includes sharing threat intelligence and developing strategies to combat cyber threats (msft_ddr.pdf).\n",
|
||||
"4. **Adopt Azure's Blockchain Services**: Small businesses can use Azure Blockchain Workbench to develop and deploy blockchain applications. This can be particularly useful for businesses looking to enhance transparency, security, and efficiency in their operations.\n",
|
||||
"\n",
|
||||
"Overall, Microsoft prioritizes secure, compliant, and effective communication and collaboration with clients through its suite of tools and services, while also emphasizing the importance of partnerships and collective efforts in cybersecurity.\n",
|
||||
"5. **Utilize Azure's Global Reach**: With Azure's extensive global infrastructure, small businesses can expand their reach to new markets and customers. Azure's regional presence allows businesses to deploy applications closer to their customers, improving performance and user experience.\n",
|
||||
"\n",
|
||||
"6. **Engage in Azure's Training and Certification Programs**: Small businesses can benefit from Azure's training and certification programs to upskill their workforce, ensuring they have the necessary expertise to leverage Azure's capabilities effectively.\n",
|
||||
"\n",
|
||||
"For a more detailed and specific approach outlined in an RFP, it would be necessary to access the actual RFP document or consult with the issuing organization.\n",
|
||||
"Running step combine_answers\n",
|
||||
"Step combine_answers produced no event\n",
|
||||
"Step handle_question produced event QuestionAnsweredEvent\n",
|
||||
">> Asked question: What are the terms and conditions related to confidentiality and data protection?\n",
|
||||
">> Got response: The terms and conditions related to confidentiality and data protection for Microsoft Azure and its services are outlined in various documents and reports. Here are some key points:\n",
|
||||
">> Asked question: What are the RFP's requirements for compliance with federal regulations and standards, and how does Microsoft Azure ensure adherence to these regulations?\n",
|
||||
">> Got response: The RFP's requirements for compliance with federal regulations and standards, and how Microsoft Azure ensures adherence to these regulations, can be summarized as follows:\n",
|
||||
"\n",
|
||||
"1. **Compliance and Certifications**: Microsoft Azure complies with numerous global, U.S. government, industry, and regional certifications and compliance standards, such as ISO 27001, FedRAMP, HIPAA, and GDPR. These standards ensure that Azure services meet stringent data protection and confidentiality requirements.\n",
|
||||
"1. **Compliance with Federal Regulations and Standards**:\n",
|
||||
" - Microsoft Azure Government is specifically designed to meet the stringent compliance requirements of U.S. government entities. It provides a secure and compliant cloud environment that is operated by screened U.S. persons and is physically isolated from other Azure instances.\n",
|
||||
" - Azure Government has achieved compliance with several federal standards, including FedRAMP High, which is a standardized approach to security assessment, authorization, and continuous monitoring for cloud services used by the federal government.\n",
|
||||
" - Azure Government also covers compliance with the Criminal Justice Information Services (CJIS) standards in 26 states, ensuring that law enforcement agencies can securely use cloud services.\n",
|
||||
"\n",
|
||||
"2. **Data Sovereignty and Privacy**: Azure Government ensures data sovereignty by keeping data within the U.S. and is operated by screened U.S. persons. Microsoft emphasizes security, privacy, control, compliance, transparency, and reliability in its services.\n",
|
||||
"2. **Ensuring Adherence to Regulations**:\n",
|
||||
" - Microsoft Azure has established the Azure Trust Center, which provides information on compliance programs and certifications, such as ISO 27001:2005 and HIPAA. This center helps organizations understand how Azure meets various compliance requirements.\n",
|
||||
" - Azure Government offers dedicated physical networks and geo-replication between locations to ensure data sovereignty and security.\n",
|
||||
" - Microsoft Azure has received the Joint Authorization Board (JAB) Provisional Authority to Operate (P-ATO) under FedRAMP guidelines, which demonstrates its commitment to maintaining high security and compliance standards.\n",
|
||||
"\n",
|
||||
"3. **Legal and Regulatory Challenges**: Microsoft faces evolving legal requirements related to data protection, such as the EU General Data Protection Regulation (GDPR) and other international data privacy laws. These regulations impose compliance obligations on Microsoft regarding the handling of personal data.\n",
|
||||
"\n",
|
||||
"4. **Security Measures**: Microsoft is committed to protecting user data from cyberattacks and unauthorized access. This includes designing products that prioritize security, privacy, integrity, and reliability. Microsoft also opposes cyberattacks on innocent citizens and enterprises.\n",
|
||||
"\n",
|
||||
"5. **Data Breaches and Liability**: Despite efforts to secure data, Microsoft acknowledges the risk of data breaches and the potential for legal exposure. The company advocates for transparency in government data requests and emphasizes the importance of protecting customer data.\n",
|
||||
"\n",
|
||||
"6. **Partnerships and Collaboration**: Microsoft collaborates with various organizations to enhance cybersecurity and protect against cyber threats. This includes initiatives like the Cybersecurity Tech Accord, which aims to protect users from cyber threats and promote responsible nation-state behavior.\n",
|
||||
"\n",
|
||||
"These points highlight Microsoft's commitment to confidentiality and data protection through compliance with international standards, legal obligations, and proactive security measures.\n",
|
||||
"Overall, Microsoft Azure ensures adherence to federal regulations and standards by providing a secure, compliant, and dedicated cloud environment tailored for U.S. government needs, supported by a comprehensive compliance framework and continuous monitoring.\n",
|
||||
"Running step combine_answers\n",
|
||||
"Step combine_answers produced no event\n",
|
||||
"Step handle_question produced event QuestionAnsweredEvent\n",
|
||||
">> Asked question: What are the risks associated with the project, and how will they be managed?\n",
|
||||
">> Got response: The risks associated with Microsoft's projects, as outlined in the retrieved documents, include a variety of operational, economic, legal, regulatory, and cybersecurity risks. Here's a summary of the key risks and how they are managed:\n",
|
||||
">> Asked question: What are the RFP's criteria for evaluating cloud service automation, and how can Microsoft Azure's automation tools and APIs fulfill these criteria?\n",
|
||||
">> Got response: To address the question of RFP criteria for evaluating cloud service automation and how Microsoft Azure's automation tools and APIs fulfill these criteria, let's break it down into two parts:\n",
|
||||
"\n",
|
||||
"1. **Operational Risks**:\n",
|
||||
" - **Infrastructure and Service Disruptions**: Microsoft faces risks related to outages, data losses, and disruptions due to inadequate operations infrastructure. To manage these, Microsoft invests in building, purchasing, or leasing data centers and equipment, and upgrading technology and network infrastructure.\n",
|
||||
" - **Quality and Supply Problems**: Risks include defects in hardware and software products, and limited suppliers for certain components. Microsoft manages these risks through design, testing, warranty repairs, and maintaining a diversified supply chain.\n",
|
||||
"1. **RFP Criteria for Evaluating Cloud Service Automation:**\n",
|
||||
" - The retrieved documents did not explicitly list RFP criteria for evaluating cloud service automation. However, typical criteria might include factors such as scalability, compliance with standards, integration capabilities, ease of use, cost-effectiveness, security features, and support for various deployment models.\n",
|
||||
"\n",
|
||||
"2. **Economic and Geopolitical Risks**:\n",
|
||||
" - **Global Business Exposure**: Microsoft's international operations expose it to economic and geopolitical risks, such as currency fluctuations and political instability. The company hedges a portion of its international currency exposure and monitors global developments.\n",
|
||||
" - **Catastrophic Events**: Events like earthquakes, pandemics, or geopolitical conflicts could disrupt business operations. Microsoft emphasizes business continuity management and resilience planning.\n",
|
||||
"2. **Microsoft Azure's Automation Tools and APIs:**\n",
|
||||
" - **Azure Automation Tools:** Azure provides a range of automation tools such as Azure Functions, which support serverless computing and allow for event-driven execution without managing server resources. Azure Logic Apps enable workflow automation and integration, supporting hybrid apps and line-of-business integration.\n",
|
||||
" - **APIs and Integration:** Azure offers APIs built on REST, HTTP, and XML, allowing developers to interact with Azure services. It also provides a client-side managed class library for encapsulating functions and integrates with development environments like Microsoft Visual Studio, Git, and Eclipse.\n",
|
||||
" - **Deployment Models:** Azure supports both the classic model and the Azure Resource Manager, which allows grouping related services for easier deployment, management, and monitoring.\n",
|
||||
" - **Platform Services:** Azure's platform as a service (PaaS) offerings include App Services for web and mobile apps, SQL PaaS for scalable databases, and Azure Media Services for media processing.\n",
|
||||
" - **Security and Compliance:** Azure Government provides a dedicated physical network and compliance with standards like FedRAMP and CJIS, which are crucial for government-related cloud services.\n",
|
||||
"\n",
|
||||
"3. **Legal, Regulatory, and Litigation Risks**:\n",
|
||||
" - **Competition and Antitrust**: Microsoft is subject to scrutiny under competition laws, which may affect product design and marketing. The company engages with regulators and adapts its strategies to comply with legal requirements.\n",
|
||||
"\n",
|
||||
"4. **Cybersecurity Risks**:\n",
|
||||
" - **Supply Chain Security**: Microsoft addresses risks from supply chain attacks through supplier audits, education, and awareness training. It also collaborates internationally to develop security norms and regulations.\n",
|
||||
" - **Infrastructure Resilience**: Microsoft invests in cybersecurity measures, such as Zero Trust principles, secure boot for build machines, and threat modeling for its DevOps environment.\n",
|
||||
"\n",
|
||||
"5. **Intellectual Property Risks**:\n",
|
||||
" - **Protection and Utilization**: Microsoft faces challenges in protecting its intellectual property globally. The company manages these risks through licensing agreements, legal actions, and engagement with open-source software.\n",
|
||||
"\n",
|
||||
"6. **General Risks**:\n",
|
||||
" - **Reputation and Brand Damage**: Risks include product safety issues, data breaches, and public scrutiny. Microsoft manages these through proactive communication, compliance, and maintaining high standards for product quality and safety.\n",
|
||||
"\n",
|
||||
"Overall, Microsoft employs a combination of strategic investments, regulatory compliance, risk management frameworks, and technological innovations to manage these risks effectively.\n",
|
||||
"In summary, while the specific RFP criteria were not detailed in the retrieved documents, Microsoft Azure's automation tools and APIs offer a comprehensive suite of services that can meet various typical criteria for cloud service automation, including scalability, integration, and compliance.\n",
|
||||
"Running step combine_answers\n",
|
||||
"Step combine_answers produced event CollectedAnswersEvent\n",
|
||||
"Running step generate_output\n",
|
||||
@@ -890,172 +999,178 @@
|
||||
"\n",
|
||||
"## Executive Summary\n",
|
||||
"\n",
|
||||
"Our proposal for the JEDI Cloud project is designed to meet the Department of Defense's requirements for a robust, secure, and scalable cloud infrastructure. We leverage our extensive experience in cloud services to deliver a solution that ensures high availability, data security, and seamless integration with existing DoD systems. Our approach emphasizes compliance with all regulatory requirements, including those related to data protection and confidentiality, while providing a flexible and cost-effective solution.\n",
|
||||
"Microsoft Azure is pleased to submit this proposal in response to the JEDI Cloud RFP # HQ0034-18-R-0077. Our proposal outlines how Microsoft Azure's comprehensive suite of cloud services meets the requirements outlined in the RFP, including high availability, security, data portability, program management, and compliance with federal regulations. We are committed to delivering a secure, scalable, and flexible cloud solution that supports the Department of Defense's mission-critical operations.\n",
|
||||
"\n",
|
||||
"## Project Scope and Objectives\n",
|
||||
"## Technical Approach\n",
|
||||
"\n",
|
||||
"The JEDI Cloud project aims to provide the Department of Defense with a comprehensive cloud infrastructure that supports both unclassified and classified data processing. The primary objectives include:\n",
|
||||
"### High Availability and Failover\n",
|
||||
"\n",
|
||||
"- Delivering Infrastructure as a Service (IaaS) and Platform as a Service (PaaS) offerings that meet the DoD's operational requirements.\n",
|
||||
"- Ensuring high availability and failover capabilities across multiple data centers.\n",
|
||||
"- Providing secure data transfer and logical isolation to protect sensitive information.\n",
|
||||
"- Supporting tactical edge operations with portable and ruggedized compute and storage solutions.\n",
|
||||
"The RFP requires high availability and failover capabilities for cloud services. Microsoft Azure offers several solutions to meet these requirements:\n",
|
||||
"\n",
|
||||
"## Deliverables\n",
|
||||
"- **Azure Site Recovery**: Provides comprehensive disaster recovery solutions, supporting site-to-Azure, any cloud, and site-to-site recovery options.\n",
|
||||
"- **Azure Kubernetes Service (AKS)**: Enables deployment of production-ready Kubernetes clusters, ensuring high availability and managing failover scenarios.\n",
|
||||
"- **Azure Functions**: Supports serverless computing architectures, contributing to high availability by automatically scaling based on demand.\n",
|
||||
"- **Azure Fabric Controller**: Maintains scalability and dependability of services, preventing failures and managing web applications, memory allocation, and load balancing.\n",
|
||||
"\n",
|
||||
"The specific deliverables for this project include:\n",
|
||||
"### Security Requirements\n",
|
||||
"\n",
|
||||
"- Unclassified and Classified IaaS and PaaS offerings.\n",
|
||||
"- Cloud Support Packages for both unclassified and classified environments.\n",
|
||||
"- A comprehensive Portability Plan and Portability Test to ensure data and application mobility.\n",
|
||||
"- Program Management Support for the Cloud Computing Program Office (CCPO).\n",
|
||||
"Microsoft Azure Government provides robust compliance standards and security features to address the JEDI Cloud's security requirements:\n",
|
||||
"\n",
|
||||
"## Budget and Payment Terms\n",
|
||||
"- **Compliance Standards**: Azure Government complies with FedRAMP Moderate and High JAB P-ATO, DoD DISA SRG Levels 2, 4, and 5, NIST SP 800-171, FIPS 140-2, CJIS, ITAR, and IRS 1075.\n",
|
||||
"- **Security Features**: Includes dedicated physical networks, ExpressRoute for secure connectivity, Azure Active Directory for identity management, and integrated management and security features.\n",
|
||||
"\n",
|
||||
"The maximum contract limit for the JEDI Cloud ID/IQ Contract is $10,000,000,000.00, with a minimum guaranteed award amount of $1,000,000.00. All task orders will be firm-fixed price, and payment terms will be structured according to the delivery and acceptance of specific contract line items.\n",
|
||||
"### Data Portability and Interoperability\n",
|
||||
"\n",
|
||||
"## Timeline and Milestones\n",
|
||||
"Microsoft Azure facilitates data portability and interoperability through:\n",
|
||||
"\n",
|
||||
"The project is structured with a two-year base ordering period, followed by two three-year option periods and one two-year option period, for a total potential duration of 10 years. Key milestones include:\n",
|
||||
"- **Data Management Services**: Azure Data Explorer, Azure Data Factory, and Azure Synapse Analytics enable seamless data movement and transformation.\n",
|
||||
"- **Hybrid Cloud Architecture**: Supports consistent identity, integrated management, and a consistent data platform across environments.\n",
|
||||
"- **Storage Services**: Azure Blob Storage, Azure Table Storage, and Azure Queue Storage provide APIs for accessing and managing data.\n",
|
||||
"\n",
|
||||
"- Initial setup and configuration of cloud infrastructure within the first six months.\n",
|
||||
"- Completion of the Portability Plan and Test within the first year.\n",
|
||||
"- Full operational capability for tactical edge solutions by the end of the second year.\n",
|
||||
"### Program Management and Oversight\n",
|
||||
"\n",
|
||||
"## Evaluation Criteria\n",
|
||||
"Microsoft Azure's management tools align with typical RFP expectations for program management and oversight:\n",
|
||||
"\n",
|
||||
"Proposals will be evaluated based on the following criteria:\n",
|
||||
"- **Azure Resource Manager**: Facilitates efficient resource management and monitoring.\n",
|
||||
"- **Azure Portal**: Provides a web-based interface for managing Azure services.\n",
|
||||
"- **Compliance and Security**: Adheres to numerous global, US government, industry, and regional standards.\n",
|
||||
"\n",
|
||||
"1. Technical Merit: The feasibility and innovation of the proposed solution.\n",
|
||||
"2. Cost-Effectiveness: The reasonableness and justification of the proposed budget.\n",
|
||||
"3. Experience and Qualifications: The expertise and past performance of the project team.\n",
|
||||
"4. Compliance and Risk Management: Adherence to regulatory requirements and risk mitigation strategies.\n",
|
||||
"5. Impact and Benefits: The potential positive outcomes for the DoD.\n",
|
||||
"### Logical Isolation and Secure Data Transfer\n",
|
||||
"\n",
|
||||
"## Project Team Qualifications\n",
|
||||
"Microsoft Azure ensures logical isolation and secure data transfer through:\n",
|
||||
"\n",
|
||||
"Our project team comprises highly qualified professionals with extensive experience in cloud services, cybersecurity, and program management. Key team members hold relevant certifications, such as Azure Developer Associate and Azure Security Engineer Associate, ensuring expertise in delivering secure and efficient cloud solutions.\n",
|
||||
"- **Physical and Logical Isolation**: Azure Government offers a physically isolated cloud environment for U.S. government entities.\n",
|
||||
"- **Identity and Access Management**: Azure Active Directory provides secure access control.\n",
|
||||
"- **Data Protection**: Azure Information Protection safeguards sensitive information.\n",
|
||||
"\n",
|
||||
"## Project Management and Reporting\n",
|
||||
"### Tactical Edge Capabilities\n",
|
||||
"\n",
|
||||
"We will implement a robust project management framework that includes:\n",
|
||||
"Microsoft Azure supports operations in communication-degraded or disconnected environments with:\n",
|
||||
"\n",
|
||||
"- Regular progress reports and performance metrics to ensure transparency and accountability.\n",
|
||||
"- Compliance with all security and data protection standards, including CJIS and FedRAMP.\n",
|
||||
"- A Quality Assurance Surveillance Plan (QASP) to monitor and maintain performance metrics.\n",
|
||||
"- **Azure IoT Edge**: Deploys cloud intelligence locally on IoT edge devices.\n",
|
||||
"- **Azure Orbital**: Provides connectivity to remote locations using satellite data.\n",
|
||||
"- **Azure Stack HCI**: Allows running virtualized workloads on-premises, connected to Azure for cloud services.\n",
|
||||
"\n",
|
||||
"## Communication and Collaboration\n",
|
||||
"### Integration of Third-Party Services\n",
|
||||
"\n",
|
||||
"We prioritize effective communication and collaboration with the DoD through:\n",
|
||||
"Microsoft Azure supports integration of third-party services through:\n",
|
||||
"\n",
|
||||
"- Regular meetings and updates to ensure alignment with project objectives.\n",
|
||||
"- Use of Microsoft Teams and other collaboration tools to facilitate seamless interaction.\n",
|
||||
"- A dedicated point of contact for all project-related inquiries and coordination.\n",
|
||||
"- **Azure Marketplace**: Offers a platform for finding, trying, and purchasing applications and services.\n",
|
||||
"- **APIs and Integration**: Provides REST and SDK APIs for storing and accessing data.\n",
|
||||
"\n",
|
||||
"## Confidentiality and Data Protection\n",
|
||||
"### Small Business Participation\n",
|
||||
"\n",
|
||||
"We are committed to maintaining the highest standards of confidentiality and data protection, as outlined in our compliance with ISO 27001, GDPR, and other relevant standards. Our approach includes:\n",
|
||||
"Microsoft Azure encourages small business participation by:\n",
|
||||
"\n",
|
||||
"- Secure data storage and transfer protocols.\n",
|
||||
"- Regular audits and assessments to ensure compliance with all regulatory requirements.\n",
|
||||
"- A comprehensive incident response plan to address any potential data breaches.\n",
|
||||
"- **Utilizing Azure's Platform Services**: Reducing costs and complexity for small businesses.\n",
|
||||
"- **Leveraging Azure IoT and AI Capabilities**: Developing smart solutions.\n",
|
||||
"- **Participating in Azure's Partner Network**: Gaining access to resources and collaboration opportunities.\n",
|
||||
"\n",
|
||||
"## Risk Management\n",
|
||||
"### Compliance with Federal Regulations\n",
|
||||
"\n",
|
||||
"We have identified key risks associated with the project, including operational disruptions, cybersecurity threats, and compliance challenges. Our risk management strategy includes:\n",
|
||||
"Microsoft Azure ensures adherence to federal regulations through:\n",
|
||||
"\n",
|
||||
"- Investing in infrastructure resilience and cybersecurity measures.\n",
|
||||
"- Engaging with regulators to ensure compliance with all legal requirements.\n",
|
||||
"- Implementing a robust supply chain management process to mitigate risks related to hardware and software components.\n",
|
||||
"- **Azure Trust Center**: Provides information on compliance programs and certifications.\n",
|
||||
"- **FedRAMP Compliance**: Achieved JAB Provisional Authority to Operate under FedRAMP guidelines.\n",
|
||||
"\n",
|
||||
"### Cloud Service Automation\n",
|
||||
"\n",
|
||||
"Microsoft Azure's automation tools fulfill cloud service automation criteria with:\n",
|
||||
"\n",
|
||||
"- **Azure Automation Tools**: Azure Functions and Logic Apps support serverless computing and workflow automation.\n",
|
||||
"- **APIs and Integration**: Offers APIs for interacting with Azure services.\n",
|
||||
"\n",
|
||||
"## Conclusion\n",
|
||||
"\n",
|
||||
"Our proposal for the JEDI Cloud project offers a comprehensive solution that meets the Department of Defense's requirements for a secure, scalable, and cost-effective cloud infrastructure. We are committed to delivering exceptional value and ensuring the success of the JEDI Cloud initiative.Step generate_output produced event StopEvent\n",
|
||||
"Microsoft Azure is committed to providing a secure, scalable, and flexible cloud solution that meets the JEDI Cloud RFP requirements. Our comprehensive suite of services, compliance with federal regulations, and support for small business participation make us the ideal partner for the Department of Defense's cloud needs. We look forward to the opportunity to support the DoD's mission-critical operations with our innovative cloud solutions.Step generate_output produced event StopEvent\n",
|
||||
"# Response to JEDI Cloud RFP # HQ0034-18-R-0077\n",
|
||||
"\n",
|
||||
"## Executive Summary\n",
|
||||
"\n",
|
||||
"Our proposal for the JEDI Cloud project is designed to meet the Department of Defense's requirements for a robust, secure, and scalable cloud infrastructure. We leverage our extensive experience in cloud services to deliver a solution that ensures high availability, data security, and seamless integration with existing DoD systems. Our approach emphasizes compliance with all regulatory requirements, including those related to data protection and confidentiality, while providing a flexible and cost-effective solution.\n",
|
||||
"Microsoft Azure is pleased to submit this proposal in response to the JEDI Cloud RFP # HQ0034-18-R-0077. Our proposal outlines how Microsoft Azure's comprehensive suite of cloud services meets the requirements outlined in the RFP, including high availability, security, data portability, program management, and compliance with federal regulations. We are committed to delivering a secure, scalable, and flexible cloud solution that supports the Department of Defense's mission-critical operations.\n",
|
||||
"\n",
|
||||
"## Project Scope and Objectives\n",
|
||||
"## Technical Approach\n",
|
||||
"\n",
|
||||
"The JEDI Cloud project aims to provide the Department of Defense with a comprehensive cloud infrastructure that supports both unclassified and classified data processing. The primary objectives include:\n",
|
||||
"### High Availability and Failover\n",
|
||||
"\n",
|
||||
"- Delivering Infrastructure as a Service (IaaS) and Platform as a Service (PaaS) offerings that meet the DoD's operational requirements.\n",
|
||||
"- Ensuring high availability and failover capabilities across multiple data centers.\n",
|
||||
"- Providing secure data transfer and logical isolation to protect sensitive information.\n",
|
||||
"- Supporting tactical edge operations with portable and ruggedized compute and storage solutions.\n",
|
||||
"The RFP requires high availability and failover capabilities for cloud services. Microsoft Azure offers several solutions to meet these requirements:\n",
|
||||
"\n",
|
||||
"## Deliverables\n",
|
||||
"- **Azure Site Recovery**: Provides comprehensive disaster recovery solutions, supporting site-to-Azure, any cloud, and site-to-site recovery options.\n",
|
||||
"- **Azure Kubernetes Service (AKS)**: Enables deployment of production-ready Kubernetes clusters, ensuring high availability and managing failover scenarios.\n",
|
||||
"- **Azure Functions**: Supports serverless computing architectures, contributing to high availability by automatically scaling based on demand.\n",
|
||||
"- **Azure Fabric Controller**: Maintains scalability and dependability of services, preventing failures and managing web applications, memory allocation, and load balancing.\n",
|
||||
"\n",
|
||||
"The specific deliverables for this project include:\n",
|
||||
"### Security Requirements\n",
|
||||
"\n",
|
||||
"- Unclassified and Classified IaaS and PaaS offerings.\n",
|
||||
"- Cloud Support Packages for both unclassified and classified environments.\n",
|
||||
"- A comprehensive Portability Plan and Portability Test to ensure data and application mobility.\n",
|
||||
"- Program Management Support for the Cloud Computing Program Office (CCPO).\n",
|
||||
"Microsoft Azure Government provides robust compliance standards and security features to address the JEDI Cloud's security requirements:\n",
|
||||
"\n",
|
||||
"## Budget and Payment Terms\n",
|
||||
"- **Compliance Standards**: Azure Government complies with FedRAMP Moderate and High JAB P-ATO, DoD DISA SRG Levels 2, 4, and 5, NIST SP 800-171, FIPS 140-2, CJIS, ITAR, and IRS 1075.\n",
|
||||
"- **Security Features**: Includes dedicated physical networks, ExpressRoute for secure connectivity, Azure Active Directory for identity management, and integrated management and security features.\n",
|
||||
"\n",
|
||||
"The maximum contract limit for the JEDI Cloud ID/IQ Contract is $10,000,000,000.00, with a minimum guaranteed award amount of $1,000,000.00. All task orders will be firm-fixed price, and payment terms will be structured according to the delivery and acceptance of specific contract line items.\n",
|
||||
"### Data Portability and Interoperability\n",
|
||||
"\n",
|
||||
"## Timeline and Milestones\n",
|
||||
"Microsoft Azure facilitates data portability and interoperability through:\n",
|
||||
"\n",
|
||||
"The project is structured with a two-year base ordering period, followed by two three-year option periods and one two-year option period, for a total potential duration of 10 years. Key milestones include:\n",
|
||||
"- **Data Management Services**: Azure Data Explorer, Azure Data Factory, and Azure Synapse Analytics enable seamless data movement and transformation.\n",
|
||||
"- **Hybrid Cloud Architecture**: Supports consistent identity, integrated management, and a consistent data platform across environments.\n",
|
||||
"- **Storage Services**: Azure Blob Storage, Azure Table Storage, and Azure Queue Storage provide APIs for accessing and managing data.\n",
|
||||
"\n",
|
||||
"- Initial setup and configuration of cloud infrastructure within the first six months.\n",
|
||||
"- Completion of the Portability Plan and Test within the first year.\n",
|
||||
"- Full operational capability for tactical edge solutions by the end of the second year.\n",
|
||||
"### Program Management and Oversight\n",
|
||||
"\n",
|
||||
"## Evaluation Criteria\n",
|
||||
"Microsoft Azure's management tools align with typical RFP expectations for program management and oversight:\n",
|
||||
"\n",
|
||||
"Proposals will be evaluated based on the following criteria:\n",
|
||||
"- **Azure Resource Manager**: Facilitates efficient resource management and monitoring.\n",
|
||||
"- **Azure Portal**: Provides a web-based interface for managing Azure services.\n",
|
||||
"- **Compliance and Security**: Adheres to numerous global, US government, industry, and regional standards.\n",
|
||||
"\n",
|
||||
"1. Technical Merit: The feasibility and innovation of the proposed solution.\n",
|
||||
"2. Cost-Effectiveness: The reasonableness and justification of the proposed budget.\n",
|
||||
"3. Experience and Qualifications: The expertise and past performance of the project team.\n",
|
||||
"4. Compliance and Risk Management: Adherence to regulatory requirements and risk mitigation strategies.\n",
|
||||
"5. Impact and Benefits: The potential positive outcomes for the DoD.\n",
|
||||
"### Logical Isolation and Secure Data Transfer\n",
|
||||
"\n",
|
||||
"## Project Team Qualifications\n",
|
||||
"Microsoft Azure ensures logical isolation and secure data transfer through:\n",
|
||||
"\n",
|
||||
"Our project team comprises highly qualified professionals with extensive experience in cloud services, cybersecurity, and program management. Key team members hold relevant certifications, such as Azure Developer Associate and Azure Security Engineer Associate, ensuring expertise in delivering secure and efficient cloud solutions.\n",
|
||||
"- **Physical and Logical Isolation**: Azure Government offers a physically isolated cloud environment for U.S. government entities.\n",
|
||||
"- **Identity and Access Management**: Azure Active Directory provides secure access control.\n",
|
||||
"- **Data Protection**: Azure Information Protection safeguards sensitive information.\n",
|
||||
"\n",
|
||||
"## Project Management and Reporting\n",
|
||||
"### Tactical Edge Capabilities\n",
|
||||
"\n",
|
||||
"We will implement a robust project management framework that includes:\n",
|
||||
"Microsoft Azure supports operations in communication-degraded or disconnected environments with:\n",
|
||||
"\n",
|
||||
"- Regular progress reports and performance metrics to ensure transparency and accountability.\n",
|
||||
"- Compliance with all security and data protection standards, including CJIS and FedRAMP.\n",
|
||||
"- A Quality Assurance Surveillance Plan (QASP) to monitor and maintain performance metrics.\n",
|
||||
"- **Azure IoT Edge**: Deploys cloud intelligence locally on IoT edge devices.\n",
|
||||
"- **Azure Orbital**: Provides connectivity to remote locations using satellite data.\n",
|
||||
"- **Azure Stack HCI**: Allows running virtualized workloads on-premises, connected to Azure for cloud services.\n",
|
||||
"\n",
|
||||
"## Communication and Collaboration\n",
|
||||
"### Integration of Third-Party Services\n",
|
||||
"\n",
|
||||
"We prioritize effective communication and collaboration with the DoD through:\n",
|
||||
"Microsoft Azure supports integration of third-party services through:\n",
|
||||
"\n",
|
||||
"- Regular meetings and updates to ensure alignment with project objectives.\n",
|
||||
"- Use of Microsoft Teams and other collaboration tools to facilitate seamless interaction.\n",
|
||||
"- A dedicated point of contact for all project-related inquiries and coordination.\n",
|
||||
"- **Azure Marketplace**: Offers a platform for finding, trying, and purchasing applications and services.\n",
|
||||
"- **APIs and Integration**: Provides REST and SDK APIs for storing and accessing data.\n",
|
||||
"\n",
|
||||
"## Confidentiality and Data Protection\n",
|
||||
"### Small Business Participation\n",
|
||||
"\n",
|
||||
"We are committed to maintaining the highest standards of confidentiality and data protection, as outlined in our compliance with ISO 27001, GDPR, and other relevant standards. Our approach includes:\n",
|
||||
"Microsoft Azure encourages small business participation by:\n",
|
||||
"\n",
|
||||
"- Secure data storage and transfer protocols.\n",
|
||||
"- Regular audits and assessments to ensure compliance with all regulatory requirements.\n",
|
||||
"- A comprehensive incident response plan to address any potential data breaches.\n",
|
||||
"- **Utilizing Azure's Platform Services**: Reducing costs and complexity for small businesses.\n",
|
||||
"- **Leveraging Azure IoT and AI Capabilities**: Developing smart solutions.\n",
|
||||
"- **Participating in Azure's Partner Network**: Gaining access to resources and collaboration opportunities.\n",
|
||||
"\n",
|
||||
"## Risk Management\n",
|
||||
"### Compliance with Federal Regulations\n",
|
||||
"\n",
|
||||
"We have identified key risks associated with the project, including operational disruptions, cybersecurity threats, and compliance challenges. Our risk management strategy includes:\n",
|
||||
"Microsoft Azure ensures adherence to federal regulations through:\n",
|
||||
"\n",
|
||||
"- Investing in infrastructure resilience and cybersecurity measures.\n",
|
||||
"- Engaging with regulators to ensure compliance with all legal requirements.\n",
|
||||
"- Implementing a robust supply chain management process to mitigate risks related to hardware and software components.\n",
|
||||
"- **Azure Trust Center**: Provides information on compliance programs and certifications.\n",
|
||||
"- **FedRAMP Compliance**: Achieved JAB Provisional Authority to Operate under FedRAMP guidelines.\n",
|
||||
"\n",
|
||||
"### Cloud Service Automation\n",
|
||||
"\n",
|
||||
"Microsoft Azure's automation tools fulfill cloud service automation criteria with:\n",
|
||||
"\n",
|
||||
"- **Azure Automation Tools**: Azure Functions and Logic Apps support serverless computing and workflow automation.\n",
|
||||
"- **APIs and Integration**: Offers APIs for interacting with Azure services.\n",
|
||||
"\n",
|
||||
"## Conclusion\n",
|
||||
"\n",
|
||||
"Our proposal for the JEDI Cloud project offers a comprehensive solution that meets the Department of Defense's requirements for a secure, scalable, and cost-effective cloud infrastructure. We are committed to delivering exceptional value and ensuring the success of the JEDI Cloud initiative.\n"
|
||||
"Microsoft Azure is committed to providing a secure, scalable, and flexible cloud solution that meets the JEDI Cloud RFP requirements. Our comprehensive suite of services, compliance with federal regulations, and support for small business participation make us the ideal partner for the Department of Defense's cloud needs. We look forward to the opportunity to support the DoD's mission-critical operations with our innovative cloud solutions.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
|
||||
+462
-157
@@ -1,6 +1,6 @@
|
||||
import os
|
||||
import asyncio
|
||||
from io import TextIOWrapper
|
||||
from urllib.parse import urlparse
|
||||
|
||||
import httpx
|
||||
import mimetypes
|
||||
@@ -11,8 +11,7 @@ from contextlib import asynccontextmanager
|
||||
from io import BufferedIOBase
|
||||
|
||||
from fsspec import AbstractFileSystem
|
||||
from fsspec.spec import AbstractBufferedFile
|
||||
from llama_index.core.async_utils import run_jobs
|
||||
from llama_index.core.async_utils import asyncio_run, run_jobs
|
||||
from llama_index.core.bridge.pydantic import Field, field_validator
|
||||
from llama_index.core.constants import DEFAULT_BASE_URL
|
||||
from llama_index.core.readers.base import BasePydanticReader
|
||||
@@ -22,7 +21,6 @@ from llama_parse.utils import (
|
||||
nest_asyncio_err,
|
||||
nest_asyncio_msg,
|
||||
ResultType,
|
||||
Language,
|
||||
SUPPORTED_FILE_TYPES,
|
||||
)
|
||||
from copy import deepcopy
|
||||
@@ -37,6 +35,7 @@ _DEFAULT_SEPARATOR = "\n---\n"
|
||||
class LlamaParse(BasePydanticReader):
|
||||
"""A smart-parser for files."""
|
||||
|
||||
# Library / access specific configurations
|
||||
api_key: str = Field(
|
||||
default="",
|
||||
description="The API key for the LlamaParse API.",
|
||||
@@ -46,8 +45,20 @@ class LlamaParse(BasePydanticReader):
|
||||
default=DEFAULT_BASE_URL,
|
||||
description="The base URL of the Llama Parsing API.",
|
||||
)
|
||||
result_type: ResultType = Field(
|
||||
default=ResultType.TXT, description="The result type for the parser."
|
||||
check_interval: int = Field(
|
||||
default=1,
|
||||
description="The interval in seconds to check if the parsing is done.",
|
||||
)
|
||||
custom_client: Optional[httpx.AsyncClient] = Field(
|
||||
default=None, description="A custom HTTPX client to use for sending requests."
|
||||
)
|
||||
ignore_errors: bool = Field(
|
||||
default=True,
|
||||
description="Whether or not to ignore and skip errors raised during parsing.",
|
||||
)
|
||||
max_timeout: int = Field(
|
||||
default=2000,
|
||||
description="The maximum timeout in seconds to wait for the parsing to finish.",
|
||||
)
|
||||
num_workers: int = Field(
|
||||
default=4,
|
||||
@@ -55,120 +66,47 @@ class LlamaParse(BasePydanticReader):
|
||||
lt=10,
|
||||
description="The number of workers to use sending API requests for parsing.",
|
||||
)
|
||||
check_interval: int = Field(
|
||||
default=1,
|
||||
description="The interval in seconds to check if the parsing is done.",
|
||||
)
|
||||
max_timeout: int = Field(
|
||||
default=2000,
|
||||
description="The maximum timeout in seconds to wait for the parsing to finish.",
|
||||
)
|
||||
verbose: bool = Field(
|
||||
default=True, description="Whether to print the progress of the parsing."
|
||||
result_type: ResultType = Field(
|
||||
default=ResultType.TXT, description="The result type for the parser."
|
||||
)
|
||||
show_progress: bool = Field(
|
||||
default=True, description="Show progress when parsing multiple files."
|
||||
)
|
||||
language: Language = Field(
|
||||
default=Language.ENGLISH, description="The language of the text to parse."
|
||||
)
|
||||
parsing_instruction: Optional[str] = Field(
|
||||
default="", description="The parsing instruction for the parser."
|
||||
)
|
||||
skip_diagonal_text: Optional[bool] = Field(
|
||||
default=False,
|
||||
description="If set to true, the parser will ignore diagonal text (when the text rotation in degrees modulo 90 is not 0).",
|
||||
)
|
||||
invalidate_cache: Optional[bool] = Field(
|
||||
default=False,
|
||||
description="If set to true, the cache will be ignored and the document re-processes. All document are kept in cache for 48hours after the job was completed to avoid processing the same document twice.",
|
||||
)
|
||||
do_not_cache: Optional[bool] = Field(
|
||||
default=False,
|
||||
description="If set to true, the document will not be cached. This mean that you will be re-charged it you reprocess them as they will not be cached.",
|
||||
)
|
||||
fast_mode: Optional[bool] = Field(
|
||||
default=False,
|
||||
description="Note: Non compatible with gpt-4o. If set to true, the parser will use a faster mode to extract text from documents. This mode will skip OCR of images, and table/heading reconstruction.",
|
||||
)
|
||||
premium_mode: bool = Field(
|
||||
default=False,
|
||||
description="Use our best parser mode if set to True.",
|
||||
)
|
||||
do_not_unroll_columns: Optional[bool] = Field(
|
||||
default=False,
|
||||
description="If set to true, the parser will keep column in the text according to document layout. Reduce reconstruction accuracy, and LLM's/embedings performances in most case.",
|
||||
)
|
||||
page_separator: Optional[str] = Field(
|
||||
default=None,
|
||||
description="A templated page separator to use to split the text. If it contain `{page_number}`,it will be replaced by the next page number. If not set will the default separator '\\n---\\n' will be used.",
|
||||
)
|
||||
page_prefix: Optional[str] = Field(
|
||||
default=None,
|
||||
description="A templated prefix to add to the beginning of each page. If it contain `{page_number}`, it will be replaced by the page number.",
|
||||
)
|
||||
page_suffix: Optional[str] = Field(
|
||||
default=None,
|
||||
description="A templated suffix to add to the beginning of each page. If it contain `{page_number}`, it will be replaced by the page number.",
|
||||
)
|
||||
gpt4o_mode: bool = Field(
|
||||
default=False,
|
||||
description="Whether to use gpt-4o extract text from documents.",
|
||||
)
|
||||
gpt4o_api_key: Optional[str] = Field(
|
||||
default=None,
|
||||
description="The API key for the GPT-4o API. Lowers the cost of parsing.",
|
||||
)
|
||||
bounding_box: Optional[str] = Field(
|
||||
default=None,
|
||||
description="The bounding box to use to extract text from documents describe as a string containing the bounding box margins",
|
||||
)
|
||||
target_pages: Optional[str] = Field(
|
||||
default=None,
|
||||
description="The target pages to extract text from documents. Describe as a comma separated list of page numbers. The first page of the document is page 0",
|
||||
)
|
||||
ignore_errors: bool = Field(
|
||||
default=True,
|
||||
description="Whether or not to ignore and skip errors raised during parsing.",
|
||||
)
|
||||
split_by_page: bool = Field(
|
||||
default=True,
|
||||
description="Whether to split by page using the page separator",
|
||||
)
|
||||
vendor_multimodal_api_key: Optional[str] = Field(
|
||||
default=None,
|
||||
description="The API key for the multimodal API.",
|
||||
verbose: bool = Field(
|
||||
default=True, description="Whether to print the progress of the parsing."
|
||||
)
|
||||
use_vendor_multimodal_model: bool = Field(
|
||||
default=False,
|
||||
description="Whether to use the vendor multimodal API.",
|
||||
)
|
||||
vendor_multimodal_model_name: Optional[str] = Field(
|
||||
default=None,
|
||||
description="The model name for the vendor multimodal API.",
|
||||
)
|
||||
take_screenshot: bool = Field(
|
||||
default=False,
|
||||
description="Whether to take screenshot of each page of the document.",
|
||||
)
|
||||
custom_client: Optional[httpx.AsyncClient] = Field(
|
||||
default=None, description="A custom HTTPX client to use for sending requests."
|
||||
)
|
||||
disable_ocr: bool = Field(
|
||||
default=False,
|
||||
description="Disable the OCR on the document. LlamaParse will only extract the copyable text from the document.",
|
||||
)
|
||||
is_formatting_instruction: bool = Field(
|
||||
default=True,
|
||||
description="Allow the parsing instruction to also format the output. Disable to have a cleaner markdown output.",
|
||||
)
|
||||
annotate_links: bool = Field(
|
||||
|
||||
# Parsing specific configurations (Alphabetical order)
|
||||
annotate_links: Optional[bool] = Field(
|
||||
default=False,
|
||||
description="Annotate links found in the document to extract their URL.",
|
||||
)
|
||||
webhook_url: Optional[str] = Field(
|
||||
auto_mode: Optional[bool] = Field(
|
||||
default=False,
|
||||
description="If set to true, the parser will automatically select the best mode to extract text from documents based on the rules provide. Will use the 'accurate' default mode by default and will upgrade page that match the rule to Premium mode.",
|
||||
)
|
||||
auto_mode_trigger_on_image_in_page: Optional[bool] = Field(
|
||||
default=False,
|
||||
description="If auto_mode is set to true, the parser will upgrade the page that contain an image to Premium mode.",
|
||||
)
|
||||
auto_mode_trigger_on_table_in_page: Optional[bool] = Field(
|
||||
default=False,
|
||||
description="If auto_mode is set to true, the parser will upgrade the page that contain a table to Premium mode.",
|
||||
)
|
||||
auto_mode_trigger_on_text_in_page: Optional[str] = Field(
|
||||
default=None,
|
||||
description="A URL that needs to be called at the end of the parsing job.",
|
||||
description="If auto_mode is set to true, the parser will upgrade the page that contain the text to Premium mode.",
|
||||
)
|
||||
auto_mode_trigger_on_regexp_in_page: Optional[str] = Field(
|
||||
default=None,
|
||||
description="If auto_mode is set to true, the parser will upgrade the page that match the regexp to Premium mode.",
|
||||
)
|
||||
azure_openai_api_version: Optional[str] = Field(
|
||||
default=None, description="Azure Openai API Version"
|
||||
)
|
||||
azure_openai_deployment_name: Optional[str] = Field(
|
||||
default=None, description="Azure Openai Deployment Name"
|
||||
@@ -176,12 +114,169 @@ class LlamaParse(BasePydanticReader):
|
||||
azure_openai_endpoint: Optional[str] = Field(
|
||||
default=None, description="Azure Openai Endpoint"
|
||||
)
|
||||
azure_openai_api_version: Optional[str] = Field(
|
||||
default=None, description="Azure Openai API Version"
|
||||
)
|
||||
azure_openai_key: Optional[str] = Field(
|
||||
default=None, description="Azure Openai Key"
|
||||
)
|
||||
bbox_bottom: Optional[float] = Field(
|
||||
default=None,
|
||||
description="The bottom margin of the bounding box to use to extract text from documents expressed as a float between 0 and 1 representing the percentage of the page height.",
|
||||
)
|
||||
bbox_left: Optional[float] = Field(
|
||||
default=None,
|
||||
description="The left margin of the bounding box to use to extract text from documents expressed as a float between 0 and 1 representing the percentage of the page width.",
|
||||
)
|
||||
bbox_right: Optional[float] = Field(
|
||||
default=None,
|
||||
description="The right margin of the bounding box to use to extract text from documents expressed as a float between 0 and 1 representing the percentage of the page width.",
|
||||
)
|
||||
bbox_top: Optional[float] = Field(
|
||||
default=None,
|
||||
description="The top margin of the bounding box to use to extract text from documents expressed as a float between 0 and 1 representing the percentage of the page height.",
|
||||
)
|
||||
continuous_mode: Optional[bool] = Field(
|
||||
default=False,
|
||||
description="Parse documents continuously, leading to better results on documents where tables span across two pages.",
|
||||
)
|
||||
disable_ocr: Optional[bool] = Field(
|
||||
default=False,
|
||||
description="Disable the OCR on the document. LlamaParse will only extract the copyable text from the document.",
|
||||
)
|
||||
disable_image_extraction: Optional[bool] = Field(
|
||||
default=False,
|
||||
description="If set to true, the parser will not extract images from the document. Make the parser faster.",
|
||||
)
|
||||
do_not_cache: Optional[bool] = Field(
|
||||
default=False,
|
||||
description="If set to true, the document will not be cached. This mean that you will be re-charged it you reprocess them as they will not be cached.",
|
||||
)
|
||||
do_not_unroll_columns: Optional[bool] = Field(
|
||||
default=False,
|
||||
description="If set to true, the parser will keep column in the text according to document layout. Reduce reconstruction accuracy, and LLM's/embedings performances in most case.",
|
||||
)
|
||||
extract_charts: Optional[bool] = Field(
|
||||
default=False,
|
||||
description="If set to true, the parser will extract/tag charts from the document.",
|
||||
)
|
||||
fast_mode: Optional[bool] = Field(
|
||||
default=False,
|
||||
description="Note: Non compatible with gpt-4o. If set to true, the parser will use a faster mode to extract text from documents. This mode will skip OCR of images, and table/heading reconstruction.",
|
||||
)
|
||||
guess_xlsx_sheet_names: Optional[bool] = Field(
|
||||
default=False,
|
||||
description="Whether to guess the sheet names of the xlsx file.",
|
||||
)
|
||||
html_make_all_elements_visible: Optional[bool] = Field(
|
||||
default=False,
|
||||
description="If set to true, when parsing HTML the parser will consider all elements display not element as display block.",
|
||||
)
|
||||
html_remove_fixed_elements: Optional[bool] = Field(
|
||||
default=False,
|
||||
description="If set to true, when parsing HTML the parser will remove fixed elements. Useful to hide cookie banners.",
|
||||
)
|
||||
html_remove_navigation_elements: Optional[bool] = Field(
|
||||
default=False,
|
||||
description="If set to true, when parsing HTML the parser will remove navigation elements. Useful to hide menus, header, footer.",
|
||||
)
|
||||
http_proxy: Optional[str] = Field(
|
||||
default=None,
|
||||
description="(optional) If set with input_url will use the specified http proxy to download the file.",
|
||||
)
|
||||
invalidate_cache: Optional[bool] = Field(
|
||||
default=False,
|
||||
description="If set to true, the cache will be ignored and the document re-processes. All document are kept in cache for 48hours after the job was completed to avoid processing the same document twice.",
|
||||
)
|
||||
is_formatting_instruction: Optional[bool] = Field(
|
||||
default=False,
|
||||
description="Allow the parsing instruction to also format the output. Disable to have a cleaner markdown output.",
|
||||
)
|
||||
language: Optional[str] = Field(
|
||||
default="en", description="The language of the text to parse."
|
||||
)
|
||||
max_pages: Optional[int] = Field(
|
||||
default=None,
|
||||
description="The maximum number of pages to extract text from documents. If set to 0 or not set, all pages will be that should be extracted will be extracted (can work in combination with targetPages).",
|
||||
)
|
||||
output_pdf_of_document: Optional[bool] = Field(
|
||||
default=False,
|
||||
description="If set to true, the parser will also output a PDF of the document. (except for spreadsheets)",
|
||||
)
|
||||
output_s3_path_prefix: Optional[str] = Field(
|
||||
default=None,
|
||||
description="An S3 path prefix to store the output of the parsing job. If set, the parser will upload the output to S3. The bucket need to be accessible from the LlamaIndex organization.",
|
||||
)
|
||||
page_prefix: Optional[str] = Field(
|
||||
default=None,
|
||||
description="A templated prefix to add to the beginning of each page. If it contain `{page_number}`, it will be replaced by the page number.",
|
||||
)
|
||||
page_separator: Optional[str] = Field(
|
||||
default=None,
|
||||
description="A templated page separator to use to split the text. If it contain `{page_number}`,it will be replaced by the next page number. If not set will the default separator '\\n---\\n' will be used.",
|
||||
)
|
||||
page_suffix: Optional[str] = Field(
|
||||
default=None,
|
||||
description="A templated suffix to add to the beginning of each page. If it contain `{page_number}`, it will be replaced by the page number.",
|
||||
)
|
||||
parsing_instruction: Optional[str] = Field(
|
||||
default="", description="The parsing instruction for the parser."
|
||||
)
|
||||
premium_mode: Optional[bool] = Field(
|
||||
default=False,
|
||||
description="Use our best parser mode if set to True.",
|
||||
)
|
||||
skip_diagonal_text: Optional[bool] = Field(
|
||||
default=False,
|
||||
description="If set to true, the parser will ignore diagonal text (when the text rotation in degrees modulo 90 is not 0).",
|
||||
)
|
||||
structured_output: Optional[bool] = Field(
|
||||
default=False,
|
||||
description="If set to true, the parser will output structured data based on the provided JSON Schema.",
|
||||
)
|
||||
structured_output_json_schema: Optional[str] = Field(
|
||||
default=None,
|
||||
description="A JSON Schema to use to structure the output of the parsing job. If set, the parser will output structured data based on the provided JSON Schema.",
|
||||
)
|
||||
structured_output_json_schema_name: Optional[str] = Field(
|
||||
default=None,
|
||||
description="The named JSON Schema to use to structure the output of the parsing job. For convenience / testing, LlamaParse provides a few named JSON Schema that can be used directly. Use 'imFeelingLucky' to let llamaParse dream the schema.",
|
||||
)
|
||||
take_screenshot: Optional[bool] = Field(
|
||||
default=False,
|
||||
description="Whether to take screenshot of each page of the document.",
|
||||
)
|
||||
target_pages: Optional[str] = Field(
|
||||
default=None,
|
||||
description="The target pages to extract text from documents. Describe as a comma separated list of page numbers. The first page of the document is page 0",
|
||||
)
|
||||
use_vendor_multimodal_model: Optional[bool] = Field(
|
||||
default=False,
|
||||
description="Whether to use the vendor multimodal API.",
|
||||
)
|
||||
vendor_multimodal_api_key: Optional[str] = Field(
|
||||
default=None,
|
||||
description="The API key for the multimodal API.",
|
||||
)
|
||||
vendor_multimodal_model_name: Optional[str] = Field(
|
||||
default=None,
|
||||
description="The model name for the vendor multimodal API.",
|
||||
)
|
||||
webhook_url: Optional[str] = Field(
|
||||
default=None,
|
||||
description="A URL that needs to be called at the end of the parsing job.",
|
||||
)
|
||||
|
||||
# Deprecated
|
||||
bounding_box: Optional[str] = Field(
|
||||
default=None,
|
||||
description="The bounding box to use to extract text from documents describe as a string containing the bounding box margins",
|
||||
)
|
||||
gpt4o_mode: Optional[bool] = Field(
|
||||
default=False,
|
||||
description="Whether to use gpt-4o extract text from documents.",
|
||||
)
|
||||
gpt4o_api_key: Optional[str] = Field(
|
||||
default=None,
|
||||
description="The API key for the GPT-4o API. Lowers the cost of parsing.",
|
||||
)
|
||||
|
||||
@field_validator("api_key", mode="before", check_fields=True)
|
||||
@classmethod
|
||||
@@ -213,6 +308,38 @@ class LlamaParse(BasePydanticReader):
|
||||
async with httpx.AsyncClient(timeout=self.max_timeout) as client:
|
||||
yield client
|
||||
|
||||
def _is_input_url(self, file_path: FileInput) -> bool:
|
||||
"""Check if the input is a valid URL.
|
||||
|
||||
This method checks for:
|
||||
- Proper URL scheme (http/https)
|
||||
- Valid URL structure
|
||||
- Network location (domain)
|
||||
"""
|
||||
if not isinstance(file_path, str):
|
||||
return False
|
||||
try:
|
||||
result = urlparse(file_path)
|
||||
return all(
|
||||
[
|
||||
result.scheme in ("http", "https"),
|
||||
result.netloc, # Has domain
|
||||
result.scheme, # Has scheme
|
||||
]
|
||||
)
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
def _is_s3_url(self, file_path: FileInput) -> bool:
|
||||
"""Check if the input is a valid URL.
|
||||
|
||||
This method checks for:
|
||||
- Proper S3 scheme (s3://)
|
||||
"""
|
||||
if isinstance(file_path, str):
|
||||
return file_path.startswith("s3://")
|
||||
return False
|
||||
|
||||
# upload a document and get back a job_id
|
||||
async def _create_job(
|
||||
self,
|
||||
@@ -224,6 +351,8 @@ class LlamaParse(BasePydanticReader):
|
||||
url = f"{self.base_url}/api/parsing/upload"
|
||||
files = None
|
||||
file_handle = None
|
||||
input_url = file_input if self._is_input_url(file_input) else None
|
||||
input_s3_path = file_input if self._is_s3_url(file_input) else None
|
||||
|
||||
if isinstance(file_input, (bytes, BufferedIOBase)):
|
||||
if not extra_info or "file_name" not in extra_info:
|
||||
@@ -233,6 +362,10 @@ class LlamaParse(BasePydanticReader):
|
||||
file_name = extra_info["file_name"]
|
||||
mime_type = mimetypes.guess_type(file_name)[0]
|
||||
files = {"file": (file_name, file_input, mime_type)}
|
||||
elif input_url is not None:
|
||||
files = None
|
||||
elif input_s3_path is not None:
|
||||
files = None
|
||||
elif isinstance(file_input, (str, Path, PurePosixPath, PurePath)):
|
||||
file_path = str(file_input)
|
||||
file_ext = os.path.splitext(file_path)[1].lower()
|
||||
@@ -252,59 +385,183 @@ class LlamaParse(BasePydanticReader):
|
||||
"file_input must be either a file path string, file bytes, or buffer object"
|
||||
)
|
||||
|
||||
data = {
|
||||
"language": self.language.value,
|
||||
"parsing_instruction": self.parsing_instruction,
|
||||
"invalidate_cache": self.invalidate_cache,
|
||||
"skip_diagonal_text": self.skip_diagonal_text,
|
||||
"do_not_cache": self.do_not_cache,
|
||||
"fast_mode": self.fast_mode,
|
||||
"premium_mode": self.premium_mode,
|
||||
"do_not_unroll_columns": self.do_not_unroll_columns,
|
||||
"gpt4o_mode": self.gpt4o_mode,
|
||||
"gpt4o_api_key": self.gpt4o_api_key,
|
||||
"vendor_multimodal_api_key": self.vendor_multimodal_api_key,
|
||||
"use_vendor_multimodal_model": self.use_vendor_multimodal_model,
|
||||
"vendor_multimodal_model_name": self.vendor_multimodal_model_name,
|
||||
"take_screenshot": self.take_screenshot,
|
||||
"disable_ocr": self.disable_ocr,
|
||||
"is_formatting_instruction": self.is_formatting_instruction,
|
||||
"annotate_links": self.annotate_links,
|
||||
}
|
||||
data: Dict[str, Any] = {}
|
||||
|
||||
# only send page separator to server if it is not None
|
||||
# as if a null, "" string is sent the server will then ignore the page separator instead of using the default
|
||||
if self.page_separator is not None:
|
||||
data["page_separator"] = self.page_separator
|
||||
data["from_python_package"] = True
|
||||
|
||||
if self.page_prefix is not None:
|
||||
data["page_prefix"] = self.page_prefix
|
||||
if self.annotate_links:
|
||||
data["annotate_links"] = self.annotate_links
|
||||
|
||||
if self.page_suffix is not None:
|
||||
data["page_suffix"] = self.page_suffix
|
||||
if self.auto_mode:
|
||||
data["auto_mode"] = self.auto_mode
|
||||
|
||||
if self.bounding_box is not None:
|
||||
data["bounding_box"] = self.bounding_box
|
||||
if self.auto_mode_trigger_on_image_in_page:
|
||||
data[
|
||||
"auto_mode_trigger_on_image_in_page"
|
||||
] = self.auto_mode_trigger_on_image_in_page
|
||||
|
||||
if self.target_pages is not None:
|
||||
data["target_pages"] = self.target_pages
|
||||
if self.auto_mode_trigger_on_table_in_page:
|
||||
data[
|
||||
"auto_mode_trigger_on_table_in_page"
|
||||
] = self.auto_mode_trigger_on_table_in_page
|
||||
|
||||
if self.webhook_url is not None:
|
||||
data["webhook_url"] = self.webhook_url
|
||||
if self.auto_mode_trigger_on_text_in_page is not None:
|
||||
data[
|
||||
"auto_mode_trigger_on_text_in_page"
|
||||
] = self.auto_mode_trigger_on_text_in_page
|
||||
|
||||
if self.auto_mode_trigger_on_regexp_in_page is not None:
|
||||
data[
|
||||
"auto_mode_trigger_on_regexp_in_page"
|
||||
] = self.auto_mode_trigger_on_regexp_in_page
|
||||
|
||||
if self.azure_openai_api_version is not None:
|
||||
data["azure_openai_api_version"] = self.azure_openai_api_version
|
||||
|
||||
# Azure OpenAI
|
||||
if self.azure_openai_deployment_name is not None:
|
||||
data["azure_openai_deployment_name"] = self.azure_openai_deployment_name
|
||||
|
||||
if self.azure_openai_endpoint is not None:
|
||||
data["azure_openai_endpoint"] = self.azure_openai_endpoint
|
||||
|
||||
if self.azure_openai_api_version is not None:
|
||||
data["azure_openai_api_version"] = self.azure_openai_api_version
|
||||
|
||||
if self.azure_openai_key is not None:
|
||||
data["azure_openai_key"] = self.azure_openai_key
|
||||
|
||||
if self.bbox_bottom is not None:
|
||||
data["bbox_bottom"] = self.bbox_bottom
|
||||
|
||||
if self.bbox_left is not None:
|
||||
data["bbox_left"] = self.bbox_left
|
||||
|
||||
if self.bbox_right is not None:
|
||||
data["bbox_right"] = self.bbox_right
|
||||
|
||||
if self.bbox_top is not None:
|
||||
data["bbox_top"] = self.bbox_top
|
||||
|
||||
if self.continuous_mode:
|
||||
data["continuous_mode"] = self.continuous_mode
|
||||
|
||||
if self.disable_ocr:
|
||||
data["disable_ocr"] = self.disable_ocr
|
||||
|
||||
if self.disable_image_extraction:
|
||||
data["disable_image_extraction"] = self.disable_image_extraction
|
||||
|
||||
if self.do_not_cache:
|
||||
data["do_not_cache"] = self.do_not_cache
|
||||
|
||||
if self.do_not_unroll_columns:
|
||||
data["do_not_unroll_columns"] = self.do_not_unroll_columns
|
||||
|
||||
if self.extract_charts:
|
||||
data["extract_charts"] = self.extract_charts
|
||||
|
||||
if self.fast_mode:
|
||||
data["fast_mode"] = self.fast_mode
|
||||
|
||||
if self.guess_xlsx_sheet_names:
|
||||
data["guess_xlsx_sheet_names"] = self.guess_xlsx_sheet_names
|
||||
|
||||
if self.html_make_all_elements_visible:
|
||||
data["html_make_all_elements_visible"] = self.html_make_all_elements_visible
|
||||
|
||||
if self.html_remove_fixed_elements:
|
||||
data["html_remove_fixed_elements"] = self.html_remove_fixed_elements
|
||||
|
||||
if self.html_remove_navigation_elements:
|
||||
data[
|
||||
"html_remove_navigation_elements"
|
||||
] = self.html_remove_navigation_elements
|
||||
|
||||
if self.http_proxy is not None:
|
||||
data["http_proxy"] = self.http_proxy
|
||||
|
||||
if input_url is not None:
|
||||
files = None
|
||||
data["input_url"] = str(input_url)
|
||||
|
||||
if input_s3_path is not None:
|
||||
files = None
|
||||
data["input_s3_path"] = str(input_s3_path)
|
||||
|
||||
if self.invalidate_cache:
|
||||
data["invalidate_cache"] = self.invalidate_cache
|
||||
|
||||
if self.is_formatting_instruction:
|
||||
data["is_formatting_instruction"] = self.is_formatting_instruction
|
||||
|
||||
if self.language:
|
||||
data["language"] = self.language
|
||||
|
||||
if self.max_pages is not None:
|
||||
data["max_pages"] = self.max_pages
|
||||
|
||||
if self.output_pdf_of_document:
|
||||
data["output_pdf_of_document"] = self.output_pdf_of_document
|
||||
|
||||
if self.output_s3_path_prefix is not None:
|
||||
data["output_s3_path_prefix"] = self.output_s3_path_prefix
|
||||
|
||||
if self.page_prefix is not None:
|
||||
data["page_prefix"] = self.page_prefix
|
||||
|
||||
# only send page separator to server if it is not None
|
||||
# as if a null, "" string is sent the server will then ignore the page separator instead of using the default
|
||||
if self.page_separator is not None:
|
||||
data["page_separator"] = self.page_separator
|
||||
|
||||
if self.page_suffix is not None:
|
||||
data["page_suffix"] = self.page_suffix
|
||||
|
||||
if self.parsing_instruction is not None:
|
||||
data["parsing_instruction"] = self.parsing_instruction
|
||||
|
||||
if self.premium_mode:
|
||||
data["premium_mode"] = self.premium_mode
|
||||
|
||||
if self.skip_diagonal_text:
|
||||
data["skip_diagonal_text"] = self.skip_diagonal_text
|
||||
|
||||
if self.structured_output:
|
||||
data["structured_output"] = self.structured_output
|
||||
|
||||
if self.structured_output_json_schema is not None:
|
||||
data["structured_output_json_schema"] = self.structured_output_json_schema
|
||||
|
||||
if self.structured_output_json_schema_name is not None:
|
||||
data[
|
||||
"structured_output_json_schema_name"
|
||||
] = self.structured_output_json_schema_name
|
||||
|
||||
if self.take_screenshot:
|
||||
data["take_screenshot"] = self.take_screenshot
|
||||
|
||||
if self.target_pages is not None:
|
||||
data["target_pages"] = self.target_pages
|
||||
|
||||
if self.use_vendor_multimodal_model:
|
||||
data["use_vendor_multimodal_model"] = self.use_vendor_multimodal_model
|
||||
|
||||
if self.vendor_multimodal_api_key is not None:
|
||||
data["vendor_multimodal_api_key"] = self.vendor_multimodal_api_key
|
||||
|
||||
if self.vendor_multimodal_model_name is not None:
|
||||
data["vendor_multimodal_model_name"] = self.vendor_multimodal_model_name
|
||||
|
||||
if self.webhook_url is not None:
|
||||
data["webhook_url"] = self.webhook_url
|
||||
|
||||
# Deprecated
|
||||
if self.bounding_box is not None:
|
||||
data["bounding_box"] = self.bounding_box
|
||||
|
||||
if self.gpt4o_mode:
|
||||
data["gpt4o_mode"] = self.gpt4o_mode
|
||||
|
||||
if self.gpt4o_api_key is not None:
|
||||
data["gpt4o_api_key"] = self.gpt4o_api_key
|
||||
|
||||
try:
|
||||
async with self.client_context() as client:
|
||||
response = await client.post(
|
||||
@@ -321,12 +578,6 @@ class LlamaParse(BasePydanticReader):
|
||||
if file_handle is not None:
|
||||
file_handle.close()
|
||||
|
||||
@staticmethod
|
||||
def __get_filename(f: Union[TextIOWrapper, AbstractBufferedFile]) -> str:
|
||||
if isinstance(f, TextIOWrapper):
|
||||
return f.name
|
||||
return f.full_name
|
||||
|
||||
async def _get_job_result(
|
||||
self, job_id: str, result_type: str, verbose: bool = False
|
||||
) -> Dict[str, Any]:
|
||||
@@ -420,7 +671,7 @@ class LlamaParse(BasePydanticReader):
|
||||
fs: Optional[AbstractFileSystem] = None,
|
||||
) -> List[Document]:
|
||||
"""Load data from the input path."""
|
||||
if isinstance(file_path, (str, Path, bytes, BufferedIOBase)):
|
||||
if isinstance(file_path, (str, PurePosixPath, Path, bytes, BufferedIOBase)):
|
||||
return await self._aload_data(
|
||||
file_path, extra_info=extra_info, fs=fs, verbose=self.verbose
|
||||
)
|
||||
@@ -462,7 +713,7 @@ class LlamaParse(BasePydanticReader):
|
||||
) -> List[Document]:
|
||||
"""Load data from the input path."""
|
||||
try:
|
||||
return asyncio.run(self.aload_data(file_path, extra_info, fs=fs))
|
||||
return asyncio_run(self.aload_data(file_path, extra_info, fs=fs))
|
||||
except RuntimeError as e:
|
||||
if nest_asyncio_err in str(e):
|
||||
raise RuntimeError(nest_asyncio_msg)
|
||||
@@ -529,7 +780,7 @@ class LlamaParse(BasePydanticReader):
|
||||
) -> List[dict]:
|
||||
"""Parse the input path."""
|
||||
try:
|
||||
return asyncio.run(self.aget_json(file_path, extra_info))
|
||||
return asyncio_run(self.aget_json(file_path, extra_info))
|
||||
except RuntimeError as e:
|
||||
if nest_asyncio_err in str(e):
|
||||
raise RuntimeError(nest_asyncio_msg)
|
||||
@@ -592,7 +843,61 @@ class LlamaParse(BasePydanticReader):
|
||||
def get_images(self, json_result: List[dict], download_path: str) -> List[dict]:
|
||||
"""Download images from the parsed result."""
|
||||
try:
|
||||
return asyncio.run(self.aget_images(json_result, download_path))
|
||||
return asyncio_run(self.aget_images(json_result, download_path))
|
||||
except RuntimeError as e:
|
||||
if nest_asyncio_err in str(e):
|
||||
raise RuntimeError(nest_asyncio_msg)
|
||||
else:
|
||||
raise e
|
||||
|
||||
async def aget_xlsx(
|
||||
self, json_result: List[dict], download_path: str
|
||||
) -> List[dict]:
|
||||
"""Download images from the parsed result."""
|
||||
headers = {"Authorization": f"Bearer {self.api_key}"}
|
||||
|
||||
# make the download path
|
||||
if not os.path.exists(download_path):
|
||||
os.makedirs(download_path)
|
||||
try:
|
||||
xlsx_list = []
|
||||
for result in json_result:
|
||||
job_id = result["job_id"]
|
||||
if self.verbose:
|
||||
print("> XLSX")
|
||||
|
||||
xlsx_path = os.path.join(download_path, f"{job_id}.xlsx")
|
||||
|
||||
xlsx = {}
|
||||
|
||||
xlsx["path"] = xlsx_path
|
||||
xlsx["job_id"] = job_id
|
||||
xlsx["original_file_path"] = result.get("file_path", None)
|
||||
|
||||
with open(xlsx_path, "wb") as f:
|
||||
xlsx_url = (
|
||||
f"{self.base_url}/api/parsing/job/{job_id}/result/raw/xlsx"
|
||||
)
|
||||
async with self.client_context() as client:
|
||||
res = await client.get(
|
||||
xlsx_url, headers=headers, timeout=self.max_timeout
|
||||
)
|
||||
res.raise_for_status()
|
||||
f.write(res.content)
|
||||
xlsx_list.append(xlsx)
|
||||
return xlsx_list
|
||||
|
||||
except Exception as e:
|
||||
print("Error while downloading xlsx:", e)
|
||||
if self.ignore_errors:
|
||||
return []
|
||||
else:
|
||||
raise e
|
||||
|
||||
def get_xlsx(self, json_result: List[dict], download_path: str) -> List[dict]:
|
||||
"""Download xlsx from the parsed result."""
|
||||
try:
|
||||
return asyncio_run(self.aget_xlsx(json_result, download_path))
|
||||
except RuntimeError as e:
|
||||
if nest_asyncio_err in str(e):
|
||||
raise RuntimeError(nest_asyncio_msg)
|
||||
|
||||
@@ -10,6 +10,8 @@ class ResultType(str, Enum):
|
||||
|
||||
TXT = "text"
|
||||
MD = "markdown"
|
||||
JSON = "json"
|
||||
STRUCTURED = "structured"
|
||||
|
||||
|
||||
class Language(str, Enum):
|
||||
|
||||
Generated
+989
-964
File diff suppressed because it is too large
Load Diff
+4
-2
@@ -4,7 +4,7 @@ build-backend = "poetry.core.masonry.api"
|
||||
|
||||
[tool.poetry]
|
||||
name = "llama-parse"
|
||||
version = "0.5.10"
|
||||
version = "0.5.17"
|
||||
description = "Parse files into RAG-Optimized formats."
|
||||
authors = ["Logan Markewich <logan@llamaindex.ai>"]
|
||||
license = "MIT"
|
||||
@@ -12,12 +12,14 @@ readme = "README.md"
|
||||
packages = [{include = "llama_parse"}]
|
||||
|
||||
[tool.poetry.dependencies]
|
||||
python = ">=3.8.1,<4.0"
|
||||
python = ">=3.9,<4.0"
|
||||
llama-index-core = ">=0.11.0"
|
||||
pydantic = "!=2.10"
|
||||
click = "^8.1.7"
|
||||
|
||||
[tool.poetry.group.dev.dependencies]
|
||||
pytest = "^8.0.0"
|
||||
pytest-asyncio = "*"
|
||||
ipykernel = "^6.29.0"
|
||||
|
||||
[tool.poetry.scripts]
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 347 KiB |
+77
-7
@@ -1,5 +1,6 @@
|
||||
import os
|
||||
import pytest
|
||||
import shutil
|
||||
from fsspec.implementations.local import LocalFileSystem
|
||||
from httpx import AsyncClient
|
||||
|
||||
@@ -76,13 +77,14 @@ def test_simple_page_markdown_buffer(markdown_parser: LlamaParse) -> None:
|
||||
os.environ.get("LLAMA_CLOUD_API_KEY", "") == "",
|
||||
reason="LLAMA_CLOUD_API_KEY not set",
|
||||
)
|
||||
def test_simple_page_with_custom_fs() -> None:
|
||||
@pytest.mark.asyncio
|
||||
async def test_simple_page_with_custom_fs() -> None:
|
||||
parser = LlamaParse(result_type="markdown")
|
||||
fs = LocalFileSystem()
|
||||
filepath = os.path.join(
|
||||
os.path.dirname(__file__), "test_files/attention_is_all_you_need.pdf"
|
||||
)
|
||||
result = parser.load_data(filepath, fs=fs)
|
||||
result = await parser.aload_data(filepath, fs=fs)
|
||||
assert len(result) == 1
|
||||
|
||||
|
||||
@@ -90,13 +92,14 @@ def test_simple_page_with_custom_fs() -> None:
|
||||
os.environ.get("LLAMA_CLOUD_API_KEY", "") == "",
|
||||
reason="LLAMA_CLOUD_API_KEY not set",
|
||||
)
|
||||
def test_simple_page_progress_workers() -> None:
|
||||
@pytest.mark.asyncio
|
||||
async def test_simple_page_progress_workers() -> None:
|
||||
parser = LlamaParse(result_type="markdown", show_progress=True, verbose=True)
|
||||
|
||||
filepath = os.path.join(
|
||||
os.path.dirname(__file__), "test_files/attention_is_all_you_need.pdf"
|
||||
)
|
||||
result = parser.load_data([filepath, filepath])
|
||||
result = await parser.aload_data([filepath, filepath])
|
||||
assert len(result) == 2
|
||||
assert len(result[0].text) > 0
|
||||
|
||||
@@ -107,7 +110,7 @@ def test_simple_page_progress_workers() -> None:
|
||||
filepath = os.path.join(
|
||||
os.path.dirname(__file__), "test_files/attention_is_all_you_need.pdf"
|
||||
)
|
||||
result = parser.load_data([filepath, filepath])
|
||||
result = await parser.aload_data([filepath, filepath])
|
||||
assert len(result) == 2
|
||||
assert len(result[0].text) > 0
|
||||
|
||||
@@ -116,12 +119,79 @@ def test_simple_page_progress_workers() -> None:
|
||||
os.environ.get("LLAMA_CLOUD_API_KEY", "") == "",
|
||||
reason="LLAMA_CLOUD_API_KEY not set",
|
||||
)
|
||||
def test_custom_client() -> None:
|
||||
@pytest.mark.asyncio
|
||||
async def test_custom_client() -> None:
|
||||
custom_client = AsyncClient(verify=False, timeout=10)
|
||||
parser = LlamaParse(result_type="markdown", custom_client=custom_client)
|
||||
filepath = os.path.join(
|
||||
os.path.dirname(__file__), "test_files/attention_is_all_you_need.pdf"
|
||||
)
|
||||
result = parser.load_data(filepath)
|
||||
result = await parser.aload_data(filepath)
|
||||
assert len(result) == 1
|
||||
assert len(result[0].text) > 0
|
||||
|
||||
|
||||
@pytest.mark.skipif(
|
||||
os.environ.get("LLAMA_CLOUD_API_KEY", "") == "",
|
||||
reason="LLAMA_CLOUD_API_KEY not set",
|
||||
)
|
||||
@pytest.mark.asyncio
|
||||
async def test_input_url() -> None:
|
||||
parser = LlamaParse(result_type="markdown")
|
||||
|
||||
# links to a resume example
|
||||
input_url = "https://cdn-blog.novoresume.com/articles/google-docs-resume-templates/basic-google-docs-resume.png"
|
||||
result = await parser.aload_data(input_url)
|
||||
assert len(result) == 1
|
||||
assert "your name" in result[0].text.lower()
|
||||
|
||||
|
||||
@pytest.mark.skipif(
|
||||
os.environ.get("LLAMA_CLOUD_API_KEY", "") == "",
|
||||
reason="LLAMA_CLOUD_API_KEY not set",
|
||||
)
|
||||
@pytest.mark.asyncio
|
||||
async def test_input_url_with_website_input() -> None:
|
||||
parser = LlamaParse(result_type="markdown")
|
||||
input_url = "https://www.google.com"
|
||||
result = await parser.aload_data(input_url)
|
||||
assert len(result) == 1
|
||||
assert "google" in result[0].text.lower()
|
||||
|
||||
|
||||
@pytest.mark.skipif(
|
||||
os.environ.get("LLAMA_CLOUD_API_KEY", "") == "",
|
||||
reason="LLAMA_CLOUD_API_KEY not set",
|
||||
)
|
||||
@pytest.mark.asyncio
|
||||
async def test_mixing_input_types() -> None:
|
||||
parser = LlamaParse(result_type="markdown")
|
||||
filepath = os.path.join(
|
||||
os.path.dirname(__file__), "test_files/attention_is_all_you_need.pdf"
|
||||
)
|
||||
input_url = "https://cdn-blog.novoresume.com/articles/google-docs-resume-templates/basic-google-docs-resume.png"
|
||||
result = await parser.aload_data([filepath, input_url])
|
||||
|
||||
assert len(result) == 2
|
||||
|
||||
|
||||
@pytest.mark.skipif(
|
||||
os.environ.get("LLAMA_CLOUD_API_KEY", "") == "",
|
||||
reason="LLAMA_CLOUD_API_KEY not set",
|
||||
)
|
||||
@pytest.mark.asyncio
|
||||
async def test_download_images() -> None:
|
||||
parser = LlamaParse(result_type="markdown", take_screenshot=True)
|
||||
filepath = os.path.join(
|
||||
os.path.dirname(__file__), "test_files/attention_is_all_you_need.pdf"
|
||||
)
|
||||
json_result = await parser.aget_json([filepath])
|
||||
|
||||
assert len(json_result) == 1
|
||||
assert len(json_result[0]["pages"][0]["images"]) > 0
|
||||
|
||||
download_path = os.path.join(os.path.dirname(__file__), "test_files/images")
|
||||
shutil.rmtree(download_path, ignore_errors=True)
|
||||
|
||||
await parser.aget_images(json_result, download_path)
|
||||
assert len(os.listdir(download_path)) == len(json_result[0]["pages"][0]["images"])
|
||||
|
||||
Reference in New Issue
Block a user