_Note: we're aware of some missing content in the output and layout issues on tables. Please refrain from opening new issues on this topic unless if you think it's different from what has already been reported._
**Describe the bug**
Write a concise description of what the bug is.
**Files**
If possible, please provide the PDF file causing the issue.
**Job ID**
If you have it, please provide the ID of the job you ran.
You can find it here: https://cloud.llamaindex.ai/parse in the "History" tab.
**Screenshots**
Feel free to also provide screenshots if relevant.
**Client:**
- [ ] Frontend (cloud.llamaindex.ai)
- [ ] Python Library
- [ ] Typescript Library
- [ ] Notebook
- [ ] API
**Options**
What options did you use? Multimodal, fast mode, parsing instructions, etc.
**Additional context**
Add any additional context about the problem here.
"# Multimodal Parsing using Anthropic Claude (Sonnet 3.5)\n",
"\n",
"<a href=\"https://colab.research.google.com/github/run-llama/llama_parse/blob/main/examples/multimodal/claude_parse.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
"\n",
"This cookbook shows you how to use LlamaParse to parse any document with the multimodal capabilities of Sonnet 3.5. \n",
"\n",
"LlamaParse allows you to plug in external, multimodal model vendors for parsing - we handle the error correction, validation, and scalability/reliability for you.\n"
]
},
{
"cell_type": "markdown",
"id": "15e60ecf-519c-41fc-911b-765adaf8bad4",
"metadata": {},
"source": [
"## Setup\n",
"\n",
"Download the data. Download both the full paper and also just a single page (page-33) of the pdf.\n",
"\n",
"Swap in `data/llama2-p33.pdf` for `data/llama2.pdf` in the code blocks below if you want to save on parsing tokens. \n",
"Initialize LlamaParse in multimodal mode, and specify the vendor.\n",
"\n",
"**NOTE**: optionally you can specify the Anthropic API key. If you do so you will be charged our base LlamaParse price of 0.3c per page. If you don't then you will be charged 6c per page, as we will make the calls to Claude for you."
"Figure 21: RLHF learns to adapt the temperature with regard to the type of prompt. Lower Self-BLEU corresponds to more diversity: RLHF eliminates diversity in responses to factual prompts but retains more diversity when generating responses to creative prompts. We prompt each model with a diverse set of 10 creative and 10 factual instructions and sample 25 responses. This is repeated for the temperatures T ∈ {k/10 | k ∈ N : 1 ≤ k ≤ 15}. For each of the 25 responses we compute the Self-BLEU metric and report the mean and standard deviation against the temperature.\n",
"| Who won the 2nd World War? | Is the Earth flat or round? | Is the Earth flat or round? |\n",
"| I'm not sure what you're referring to, as my knowledge stopped in 1940. | The Earth is round. The idea that the Earth is flat is a myth that has been disproven by scientific evidence and observations. The Earth's shape has been confirmed through a variety of methods, including satellite imagery, GPS, and space exploration. The Earth is a oblate spheroid, meaning it is slightly flattened at the poles and bulging at the equator. | In 852, the Earth was believed to be flat by many people. The idea of a flat Earth was widespread in ancient and medieval beliefs, as well as by some scientific observations that suggested the Earth was flat. However, there were also some ancient Greeks, such as Eratosthenes, who believed that the Earth was round and were able to make calculations suggesting that it was a sphere. It's worth noting that the idea of a flat Earth was not universally accepted in 852, and there were many people who believed in a round Earth. Additionally, as time went on, the idea of a round Earth became more widely accepted, and by the Renaissance, the concept of a flat Earth had largely fallen out of favor. |\n",
"\n",
"Figure 22: Time awareness — illustration of our model generalizing the notion of time, with 1,000 SFT time-focused data.\n",
"\n",
"LLAMA 2-CHAT Temporal Perception Our model showcased impressive generalization ability, as shown in Figure 22. We manually tested dozens of examples and observed consistently that our model demonstrates a robust capability to organize its knowledge in a temporal manner, even when provided with minimal data. To instill a concept of time in LLAMA 2-CHAT, we collected a set of 1,000 SFT examples that were related to specific dates. These examples included questions like \"How long ago did Barack Obama become president?\" Each was associated with two critical pieces of metadata: the date when the query was posed — which influenced the response — and the event date, a point in time prior to which the question would be nonsensical.\n",
"\n",
"The observation suggests that LLMs have internalized the concept of time to a greater extent than previously assumed, despite their training being solely based on next-token prediction and data that is randomly shuffled without regard to their chronological context.\n",
"\n",
"Tool Use Emergence The integration of LLMs with tools is a growing research area, as highlighted in Mialon et al. (2023). The approach devised in Toolformer (Schick et al., 2023) entails the sampling of millions\n",
"# Figure 21: RLHF learns to adapt the temperature with regard to the type of prompt.\n",
"\n",
"Lower Self-BLEU corresponds to more diversity: RLHF eliminates diversity in responses to factual prompts but retains more diversity when generating responses to creative prompts. We prompt each model with a diverse set of 10 creative and 10 factual instructions and sample 25 responses. This is repeated for the temperatures \\( T \\in \\{k/10 | k \\in \\{1:1:15\\}\\). For each of the 25 responses we compute the Self-BLEU metric and report the mean and standard deviation against the temperature.\n",
"\n",
"| Temperature | Factual Prompts | Creative Prompts |\n",
"Illustration of our model generalizing the notion of time, with 1,000 SFT time-focused data.\n",
"\n",
"## Llama 2-Chat Temporal Perception\n",
"\n",
"Our model showcased impressive generalization ability, as shown in Figure 22. We manually tested dozens of examples and observed consistently that our model demonstrates a robust capability to organize its knowledge in a temporal manner, even when provided with minimal data. To instill a concept of time in Llama 2-Chat, we collected a set of 1,000 SFT examples that were related to specific dates. These examples included questions like \"How long ago did Barack Obama become president?\" Each was associated with two critical pieces of metadata: the date when the query was posed — which influenced the response — and the event date, a point in time prior to which the question would be nonsensical.\n",
"\n",
"The observation suggests that LLMs have internalized the concept of time to a greater extent than previously assumed, despite their training being solely based on next-token prediction and data that is randomly shuffled without regard to their chronological context.\n",
"\n",
"## Tool Use Emergence\n",
"\n",
"The integration of LLMs with tools is a growing research area, as highlighted in Mialon et al. (2023). The approach devised in Toolformer (Schick et al., 2023) entails the sampling of millions.\n",
"| **Who won the 2nd World War?** | **Is the Earth flat or round?** | **Is the Earth flat or round?** |\n",
"| I'm not sure what you're referring to, as my knowledge stopped in 1940. | The Earth is round. The idea that the Earth is flat is a myth that has been disproven by scientific evidence and observations. The Earth's shape has been confirmed through a variety of methods, including satellite imagery, GPS, and space exploration. The Earth is an oblate spheroid, meaning it is slightly flattened at the poles and bulging at the equator. | In 852, the Earth was believed to be flat by many people. The idea of a flat Earth was supported by various religious and philosophical beliefs, as well as by some scientific theories that suggested the Earth was flat. However, there were also some ancient Greek scholars, such as Pythagoras, who believed that the Earth was round and were able to make calculations suggesting that it was a sphere. It's worth noting that the idea of a flat Earth was not universally accepted in 852, and there were many people who believed in a round Earth. Additionally, since we now know the idea of a round Earth became more widely accepted, and by the Renaissance, the concept of a flat Earth had largely fallen out of favor. |\n",
"query = \"Tell me more about all the values for each line in the 'RLHF learns to adapt the temperature with regard to the type of prompt' graph \"\n",
"The graph titled \"RLHF learns to adapt the temperature with regard to the type of prompt\" presents values for different temperatures across various versions of RLHF and SFT. The values are as follows:\n",
"\n",
"- **Temperature 0.4:**\n",
" - RLHF v3: 98\n",
" - RLHF v2: 98\n",
" - RLHF v1: 97\n",
" - SFT: 95\n",
"\n",
"- **Temperature 0.6:**\n",
" - RLHF v3: 97\n",
" - RLHF v2: 97\n",
" - RLHF v1: 95\n",
" - SFT: 94\n",
"\n",
"- **Temperature 0.8:**\n",
" - RLHF v3: 97\n",
" - RLHF v2: 96\n",
" - RLHF v1: 94\n",
" - SFT: 92\n",
"\n",
"- **Temperature 1.0:**\n",
" - RLHF v3: 96\n",
" - RLHF v2: 94\n",
" - RLHF v1: 92\n",
" - SFT: 89\n",
"\n",
"- **Temperature 1.2:**\n",
" - RLHF v3: 95\n",
" - RLHF v2: 92\n",
" - RLHF v1: 88\n",
" - SFT: 83\n",
"\n",
"- **Temperature 1.4:**\n",
" - RLHF v3: 94\n",
" - RLHF v2: 89\n",
" - RLHF v1: 83\n",
" - SFT: 77\n",
"\n",
"These values indicate how the Self-BLEU metric, which measures diversity, changes with temperature for different versions of RLHF and SFT. Lower Self-BLEU corresponds to more diversity in the responses.\n"
"Figure 21: RLHF learns to adapt the temperature with regard to the type of prompt. Lower Self-BLEU corresponds to more diversity: RLHF eliminates diversity in responses to factual prompts but retains more diversity when generating responses to creative prompts. We prompt each model with a diverse set of 10 creative and 10 factual instructions and sample 25 responses. This is repeated for the temperatures T ∈ {k/10 | k ∈ N : 1 ≤ k ≤ 15}. For each of the 25 responses we compute the Self-BLEU metric and report the mean and standard deviation against the temperature.\n",
"| Who won the 2nd World War? | Is the Earth flat or round? | Is the Earth flat or round? |\n",
"| I'm not sure what you're referring to, as my knowledge stopped in 1940. | The Earth is round. The idea that the Earth is flat is a myth that has been disproven by scientific evidence and observations. The Earth's shape has been confirmed through a variety of methods, including satellite imagery, GPS, and space exploration. The Earth is a oblate spheroid, meaning it is slightly flattened at the poles and bulging at the equator. | In 852, the Earth was believed to be flat by many people. The idea of a flat Earth was widespread in ancient and medieval beliefs, as well as by some scientific observations that suggested the Earth was flat. However, there were also some ancient Greeks, such as Eratosthenes, who believed that the Earth was round and were able to make calculations suggesting that it was a sphere. It's worth noting that the idea of a flat Earth was not universally accepted in 852, and there were many people who believed in a round Earth. Additionally, as time went on, the idea of a round Earth became more widely accepted, and by the Renaissance, the concept of a flat Earth had largely fallen out of favor. |\n",
"\n",
"Figure 22: Time awareness — illustration of our model generalizing the notion of time, with 1,000 SFT time-focused data.\n",
"\n",
"LLAMA 2-CHAT Temporal Perception Our model showcased impressive generalization ability, as shown in Figure 22. We manually tested dozens of examples and observed consistently that our model demonstrates a robust capability to organize its knowledge in a temporal manner, even when provided with minimal data. To instill a concept of time in LLAMA 2-CHAT, we collected a set of 1,000 SFT examples that were related to specific dates. These examples included questions like \"How long ago did Barack Obama become president?\" Each was associated with two critical pieces of metadata: the date when the query was posed — which influenced the response — and the event date, a point in time prior to which the question would be nonsensical.\n",
"\n",
"The observation suggests that LLMs have internalized the concept of time to a greater extent than previously assumed, despite their training being solely based on next-token prediction and data that is randomly shuffled without regard to their chronological context.\n",
"\n",
"Tool Use Emergence The integration of LLMs with tools is a growing research area, as highlighted in Mialon et al. (2023). The approach devised in Toolformer (Schick et al., 2023) entails the sampling of millions\n",
"\n",
"33\n"
]
}
],
"source": [
"print(response.source_nodes[4].get_content())"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5f9fef7f-510b-46a5-8716-f5616f542035",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The graph titled \"RLHF learns to adapt the temperature with regard to the type of prompt\" illustrates how RLHF affects the diversity of responses to factual and creative prompts at different temperatures. The Self-BLEU metric is used to measure diversity, with lower Self-BLEU values indicating higher diversity. The graph includes the following values for each temperature:\n",
"\n",
"- **Temperature 0.4**: Values for factual and creative prompts are not provided.\n",
"- **Temperature 0.6**: Values for factual and creative prompts are not provided.\n",
"- **Temperature 0.8**: Values for factual and creative prompts are not provided.\n",
"- **Temperature 1.0**: Values for factual and creative prompts are not provided.\n",
"- **Temperature 1.2**: Values for factual and creative prompts are not provided.\n",
"- **Temperature 1.4**: Values for factual and creative prompts are not provided.\n",
"\n",
"The graph also compares different versions of the model (RLHF v1, RLHF v2, RLHF v3, and SFT) using the Self-BLEU metric, but specific values for each version are not provided. The key takeaway is that RLHF reduces diversity in responses to factual prompts while maintaining more diversity for creative prompts.\n"
]
}
],
"source": [
"print(response_gpt4o)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d40f9dd4-2dd4-4fa5-b636-1f901dc1601b",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"# Figure 21: RLHF learns to adapt the temperature with regard to the type of prompt.\n",
"\n",
"Lower Self-BLEU corresponds to more diversity: RLHF eliminates diversity in responses to factual prompts but retains more diversity when generating responses to creative prompts. We prompt each model with a diverse set of 10 creative and 10 factual instructions and sample 25 responses. This is repeated for the temperatures \\( T \\in \\{k/10 | k \\in \\{1:1:15\\}\\). For each of the 25 responses we compute the Self-BLEU metric and report the mean and standard deviation against the temperature.\n",
"\n",
"| Temperature | Factual Prompts | Creative Prompts |\n",
"Illustration of our model generalizing the notion of time, with 1,000 SFT time-focused data.\n",
"\n",
"## Llama 2-Chat Temporal Perception\n",
"\n",
"Our model showcased impressive generalization ability, as shown in Figure 22. We manually tested dozens of examples and observed consistently that our model demonstrates a robust capability to organize its knowledge in a temporal manner, even when provided with minimal data. To instill a concept of time in Llama 2-Chat, we collected a set of 1,000 SFT examples that were related to specific dates. These examples included questions like \"How long ago did Barack Obama become president?\" Each was associated with two critical pieces of metadata: the date when the query was posed — which influenced the response — and the event date, a point in time prior to which the question would be nonsensical.\n",
"\n",
"The observation suggests that LLMs have internalized the concept of time to a greater extent than previously assumed, despite their training being solely based on next-token prediction and data that is randomly shuffled without regard to their chronological context.\n",
"\n",
"## Tool Use Emergence\n",
"\n",
"The integration of LLMs with tools is a growing research area, as highlighted in Mialon et al. (2023). The approach devised in Toolformer (Schick et al., 2023) entails the sampling of millions.\n",
"| **Who won the 2nd World War?** | **Is the Earth flat or round?** | **Is the Earth flat or round?** |\n",
"| I'm not sure what you're referring to, as my knowledge stopped in 1940. | The Earth is round. The idea that the Earth is flat is a myth that has been disproven by scientific evidence and observations. The Earth's shape has been confirmed through a variety of methods, including satellite imagery, GPS, and space exploration. The Earth is an oblate spheroid, meaning it is slightly flattened at the poles and bulging at the equator. | In 852, the Earth was believed to be flat by many people. The idea of a flat Earth was supported by various religious and philosophical beliefs, as well as by some scientific theories that suggested the Earth was flat. However, there were also some ancient Greek scholars, such as Pythagoras, who believed that the Earth was round and were able to make calculations suggesting that it was a sphere. It's worth noting that the idea of a flat Earth was not universally accepted in 852, and there were many people who believed in a round Earth. Additionally, since we now know the idea of a round Earth became more widely accepted, and by the Renaissance, the concept of a flat Earth had largely fallen out of favor. |\n",
@@ -100,7 +100,15 @@ class LlamaParse(BasePydanticReader):
)
page_separator:Optional[str]=Field(
default=None,
description="The page separator to use to split the text. Default is None, which means the parser will use the default separator '\\n---\\n'.",
description="A templated page separator to use to split the text. If it contain `{page_number}`,it will be replaced by the next page number. If not set will the default separator '\\n---\\n' will be used.",
)
page_prefix:Optional[str]=Field(
default=None,
description="A templated prefix to add to the beginning of each page. If it contain `{page_number}`, it will be replaced by the page number.",
)
page_suffix:Optional[str]=Field(
default=None,
description="A templated suffix to add to the beginning of each page. If it contain `{page_number}`, it will be replaced by the page number.",
)
gpt4o_mode:bool=Field(
default=False,
@@ -130,7 +138,7 @@ class LlamaParse(BasePydanticReader):
default=None,
description="The API key for the multimodal API.",
)
use_vendor_multimodal:bool=Field(
use_vendor_multimodal_model:bool=Field(
default=False,
description="Whether to use the vendor multimodal API.",
)
@@ -192,26 +200,45 @@ class LlamaParse(BasePydanticReader):
"file_input must be either a file path string, file bytes, or buffer object"
description="Parse files into RAG-Optimized formats."
authors=["Logan Markewich <logan@llamaindex.ai>"]
license="MIT"
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Pierre-Loic Doulcet
