run-llama/llama_cloud_services
1
0
Fork 0
mirror of https://github.com/run-llama/llama_cloud_services.git synced 2026-08-27 19:30:01 -04:00
Code Issues 352 Packages Projects Releases 10 Wiki Activity

LLamaparse could not extract figures from one pdf file despite good pdf quality (edge-case) #366

New Issue
Open
opened 2026-02-16 00:17:38 -05:00 by yindo · 16 comments
yindo commented 2026-02-16 00:17:38 -05:00
Owner
Copy Link
Copy Source

Originally created by @haniehm on GitHub (Dec 5, 2024).

Describe the bug
I am trying to extract graphical figures and text in some pdf files. The code below, which is inspired by a helpful response of @tkcoding to another issue does work for many pdfs with figures but there is an edge case, that I can not parse despite some "prompt" engineering effort:

This is the code I used, that is mostly adopted from (https://github.com/run-llama/llama_parse/issues/317)

from llama_parse import LlamaParse
from llama_index.core import StorageContext
from llama_index.core import VectorStoreIndex
from llama_index.core.node_parser import MarkdownElementNodeParser
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding


from llama_index.core.schema import ImageDocument
from typing import List
from llama_index.core.node_parser import LlamaParseJsonNodeParser
from llama_index.core.schema import BaseNode, TextNode, Document
from llama_index.core.tools import QueryEngineTool, ToolMetadata
from llama_index.core.query_engine import SubQuestionQueryEngine
import glob
import nest_asyncio
import os
from dotenv import load_dotenv

load_dotenv()
nest_asyncio.apply()


ins = """
You are a highly proficient language model designed to convert pages from PDF into structured markdown text. Your goal is to accurately transcribe text, and identify and describe images and figures, particularly graphs and other graphical elements.

You have been tasked with creating a markdown copy of each page from the provided PDF. Each image description must include a full description of the content, a summary of the graphical object.

Maintain the sequence of all the elements.

For the following element, follow the requirement of extraction:
for Text:
   - Extract all readable text from the page.
   - Exclude any diagonal text, headers, and footers.

for Text which includes hyperlink:
    -Extract hyperlink and present it with the text
    
for Image Identification and Description:
   - Identify all images, graphs, and other graphical elements on the page.
   - If image contains wording that is hard to extract , flag it with <unidentifiable section> instead of parsing.
   - For each image, include a full description of the content in the alt text, followed by a brief summary of the graphical object.
   - If the image has a subtitle or caption, include it in the description.
   - If the image has a organisation chart , convert it into a hierachical understandable format.
   - for graph , extract the value in table form as markdown representation

for Figure Identification and Description:
   - Identify all Figures, graphs, and other graphical elements on the page. Some of those are created using alphabets, so be creative
   - If figure contains wording that is hard to extract , flag it with <unidentifiable section> instead of parsing.
   - For each figure, include a full description of the content in the alt text, followed by a brief summary of the graphical object.
   - If the figure has a subtitle or caption, include it in the description.
   - If the figure has a organisation chart , convert it into a hierachical understandable format.
   - for graph , extract the value in table form as markdown representation



    
# OUTPUT INSTRUCTIONS

- Exclude any diagonal text, headers, and footers from the output.
- For each figure, image and graph, provide a detailed description and summary.
"""

class llama_document_parser(object):
    def __init__(self,parsing_ins):
        self.parser = LlamaParse(
            verbose=True,
            ignore_errors=False,
            invalidate_cache=True,
            do_not_cache=True,
        )

    def get_image_text_nodes(self,download_path: str,json_objs: List[dict]):
        """Extract out text from images using a multimodal model."""
        image_dicts = self.parser.get_images(json_objs, download_path=download_path)
        image_documents = []
        img_text_nodes = []
        for image_dict in image_dicts:
            image_doc = ImageDocument(image_path=image_dict["path"])
            img_text_nodes.append(image_doc)
        return img_text_nodes

    def document_processing_llamaparse(self,file_name: str ,image_output_folder:str):
        """Parse document in using llamaparse and return extracted elements in json format"""
        json_objs = self.parser.get_json_result(file_name)
        json_list = json_objs[0]["pages"]
        print(json_list)
        if not os.path.exists(image_output_folder):
            os.mkdir(image_output_folder)

        image_text_nodes = self.get_image_text_nodes(image_output_folder,json_objs)
        return json_list

llama_parser = llama_document_parser(parsing_ins=ins)
# llamaparse to extract documents
json_list = llama_parser.document_processing_llamaparse(file_name="/content/drive/MyDrive/Omdena_Challenge/new_LK_tea_dataset/Guideline/with_Figure/TRISL_Guideline_02_2021e_Aug2021Intercropping.pdf",
                              image_output_folder="image_test")

Files
TRISL_Guideline_02_2021e_Aug2021Intercropping.pdf

Job ID
9d32683a-a45e-400b-8099-4f50ab29881f

Client:

  • Python Library
  • API
  • Frontend (cloud.llamaindex.ai)
  • Notebook

Additional context
I am trying to extract Figures.

Originally created by @haniehm on GitHub (Dec 5, 2024). **Describe the bug** I am trying to extract graphical figures and text in some pdf files. The code below, which is inspired by a helpful response of @tkcoding to another issue does work for many pdfs with figures but there is an edge case, that I can not parse despite some "prompt" engineering effort: This is the code I used, that is mostly adopted from (https://github.com/run-llama/llama_parse/issues/317) ```python from llama_parse import LlamaParse from llama_index.core import StorageContext from llama_index.core import VectorStoreIndex from llama_index.core.node_parser import MarkdownElementNodeParser from llama_index.llms.openai import OpenAI from llama_index.embeddings.openai import OpenAIEmbedding from llama_index.core.schema import ImageDocument from typing import List from llama_index.core.node_parser import LlamaParseJsonNodeParser from llama_index.core.schema import BaseNode, TextNode, Document from llama_index.core.tools import QueryEngineTool, ToolMetadata from llama_index.core.query_engine import SubQuestionQueryEngine import glob import nest_asyncio import os from dotenv import load_dotenv load_dotenv() nest_asyncio.apply() ins = """ You are a highly proficient language model designed to convert pages from PDF into structured markdown text. Your goal is to accurately transcribe text, and identify and describe images and figures, particularly graphs and other graphical elements. You have been tasked with creating a markdown copy of each page from the provided PDF. Each image description must include a full description of the content, a summary of the graphical object. Maintain the sequence of all the elements. For the following element, follow the requirement of extraction: for Text: - Extract all readable text from the page. - Exclude any diagonal text, headers, and footers. for Text which includes hyperlink: -Extract hyperlink and present it with the text for Image Identification and Description: - Identify all images, graphs, and other graphical elements on the page. - If image contains wording that is hard to extract , flag it with <unidentifiable section> instead of parsing. - For each image, include a full description of the content in the alt text, followed by a brief summary of the graphical object. - If the image has a subtitle or caption, include it in the description. - If the image has a organisation chart , convert it into a hierachical understandable format. - for graph , extract the value in table form as markdown representation for Figure Identification and Description: - Identify all Figures, graphs, and other graphical elements on the page. Some of those are created using alphabets, so be creative - If figure contains wording that is hard to extract , flag it with <unidentifiable section> instead of parsing. - For each figure, include a full description of the content in the alt text, followed by a brief summary of the graphical object. - If the figure has a subtitle or caption, include it in the description. - If the figure has a organisation chart , convert it into a hierachical understandable format. - for graph , extract the value in table form as markdown representation # OUTPUT INSTRUCTIONS - Exclude any diagonal text, headers, and footers from the output. - For each figure, image and graph, provide a detailed description and summary. """ class llama_document_parser(object): def __init__(self,parsing_ins): self.parser = LlamaParse( verbose=True, ignore_errors=False, invalidate_cache=True, do_not_cache=True, ) def get_image_text_nodes(self,download_path: str,json_objs: List[dict]): """Extract out text from images using a multimodal model.""" image_dicts = self.parser.get_images(json_objs, download_path=download_path) image_documents = [] img_text_nodes = [] for image_dict in image_dicts: image_doc = ImageDocument(image_path=image_dict["path"]) img_text_nodes.append(image_doc) return img_text_nodes def document_processing_llamaparse(self,file_name: str ,image_output_folder:str): """Parse document in using llamaparse and return extracted elements in json format""" json_objs = self.parser.get_json_result(file_name) json_list = json_objs[0]["pages"] print(json_list) if not os.path.exists(image_output_folder): os.mkdir(image_output_folder) image_text_nodes = self.get_image_text_nodes(image_output_folder,json_objs) return json_list llama_parser = llama_document_parser(parsing_ins=ins) # llamaparse to extract documents json_list = llama_parser.document_processing_llamaparse(file_name="/content/drive/MyDrive/Omdena_Challenge/new_LK_tea_dataset/Guideline/with_Figure/TRISL_Guideline_02_2021e_Aug2021Intercropping.pdf", image_output_folder="image_test") ``` **Files** [TRISL_Guideline_02_2021e_Aug2021Intercropping.pdf](https://github.com/user-attachments/files/18025483/TRISL_Guideline_02_2021e_Aug2021Intercropping.pdf) **Job ID** 9d32683a-a45e-400b-8099-4f50ab29881f **Client:** - Python Library - API - Frontend (cloud.llamaindex.ai) - Notebook **Additional context** I am trying to extract Figures.
yindo added the enhancement label 2026-02-16 00:17:38 -05:00
yindo commented 2026-02-16 00:17:38 -05:00
Author
Owner
Copy Link
Copy Source

@BinaryBrain commented on GitHub (Dec 6, 2024):

It looks like you don't inject the parsing instruction (parsing_instruction). Try this:

        self.parser = LlamaParse(
            verbose=True,
            ignore_errors=False,
            invalidate_cache=True,
            do_not_cache=True,
            parsing_instruction=parsing_ins,
        )
@BinaryBrain commented on GitHub (Dec 6, 2024): It looks like you don't inject the parsing instruction (`parsing_instruction`). Try this: ```py self.parser = LlamaParse( verbose=True, ignore_errors=False, invalidate_cache=True, do_not_cache=True, parsing_instruction=parsing_ins, ) ```
yindo commented 2026-02-16 00:17:38 -05:00
Author
Owner
Copy Link
Copy Source

@BinaryBrain commented on GitHub (Dec 6, 2024):

Note that we also have an option to skip_diagonal_text.

@BinaryBrain commented on GitHub (Dec 6, 2024): Note that we also have an option to [skip_diagonal_text](https://docs.cloud.llamaindex.ai/llamaparse/features/parsing_options#skip-diagonal-text).
yindo commented 2026-02-16 00:17:38 -05:00
Author
Owner
Copy Link
Copy Source

@haniehm commented on GitHub (Dec 6, 2024):

@BinaryBrain Thank you for your response. Tested with both options active and still does not work (job_id 555145c4-f0c9-4c68-a820-39220f04e069)

def __init__(self,parsing_ins): self.parser = LlamaParse( verbose=True, ignore_errors=False, invalidate_cache=True, do_not_cache=True, parsing_instruction=parsing_ins, skip_diagonal_text=True )

@haniehm commented on GitHub (Dec 6, 2024): @BinaryBrain Thank you for your response. Tested with both options active and still does not work (job_id 555145c4-f0c9-4c68-a820-39220f04e069) ` def __init__(self,parsing_ins): self.parser = LlamaParse( verbose=True, ignore_errors=False, invalidate_cache=True, do_not_cache=True, parsing_instruction=parsing_ins, skip_diagonal_text=True )`
yindo commented 2026-02-16 00:17:38 -05:00
Author
Owner
Copy Link
Copy Source

@BinaryBrain commented on GitHub (Dec 6, 2024):

Oh, sorry I missed something else:
You need to fetch the markdown and not the text by providing result_type="markdown".

        self.parser = LlamaParse(
            verbose=True,
            ignore_errors=False,
            invalidate_cache=True,
            do_not_cache=True,
            parsing_instruction=parsing_ins,
            result_type="markdown"
        )

The parsing instruction don't apply to text because they are in a raw format.
If you don't want to rerun the job, you can use this API endpoint: https://api.cloud.llamaindex.ai/api/parsing/job/9d32683a-a45e-400b-8099-4f50ab29881f/result/markdown

@BinaryBrain commented on GitHub (Dec 6, 2024): Oh, sorry I missed something else: You need to fetch the markdown and not the text by providing `result_type="markdown"`. ```py self.parser = LlamaParse( verbose=True, ignore_errors=False, invalidate_cache=True, do_not_cache=True, parsing_instruction=parsing_ins, result_type="markdown" ) ``` The parsing instruction don't apply to text because they are in a raw format. If you don't want to rerun the job, you can use this API endpoint: https://api.cloud.llamaindex.ai/api/parsing/job/9d32683a-a45e-400b-8099-4f50ab29881f/result/markdown
yindo commented 2026-02-16 00:17:38 -05:00
Author
Owner
Copy Link
Copy Source

@haniehm commented on GitHub (Dec 6, 2024):

@BinaryBrain Thank you I just tried def __init__(self,parsing_ins): self.parser = LlamaParse( verbose=True, ignore_errors=False, invalidate_cache=True, do_not_cache=True, parsing_instruction=parsing_ins, skip_diagonal_text=True, result_type="markdown" ) and still no luck (job_id ea6c8459-5b9e-4336-970a-cd12040e5142).

@haniehm commented on GitHub (Dec 6, 2024): @BinaryBrain Thank you I just tried` def __init__(self,parsing_ins): self.parser = LlamaParse( verbose=True, ignore_errors=False, invalidate_cache=True, do_not_cache=True, parsing_instruction=parsing_ins, skip_diagonal_text=True, result_type="markdown" )` and still no luck (job_id ea6c8459-5b9e-4336-970a-cd12040e5142).
yindo commented 2026-02-16 00:17:38 -05:00
Author
Owner
Copy Link
Copy Source

@BinaryBrain commented on GitHub (Dec 6, 2024):

Can you post the markdown you get?
Have you tried another mode? Premium works very well on charts and figures.

@BinaryBrain commented on GitHub (Dec 6, 2024): Can you post the markdown you get? Have you tried another mode? Premium works very well on charts and figures.
yindo commented 2026-02-16 00:17:38 -05:00
Author
Owner
Copy Link
Copy Source

@haniehm commented on GitHub (Dec 6, 2024):

response.txt you can search for "Figure 1. " and "Figure 2." and see the outcome in the markdown.
Have not tried Premium since this is a pro-bono project with 0 budget sadly.
Thanks

@haniehm commented on GitHub (Dec 6, 2024): [response.txt](https://github.com/user-attachments/files/18040931/response.txt) you can search for "Figure 1. " and "Figure 2." and see the outcome in the markdown. Have not tried Premium since this is a pro-bono project with 0 budget sadly. Thanks
yindo commented 2026-02-16 00:17:38 -05:00
Author
Owner
Copy Link
Copy Source

@tkcoding commented on GitHub (Dec 13, 2024):

@haniehm
Sorry for the last reply , if you haven find the solution.
Basically llama is still working on improving the image extraction ,specifically for images that looks like a bunch of text in an images often confuse the outcome of the extraction (extracted as bunch of text with delimiter).

Solution:
Hybrid with other framework like paddleOCR (best solution) or pymupdf (OK solution). To extract the images. Create a multi-vector retriever with images processed into text and taking markdown from llamaparse.

This will yield the best quality so far.

Note: Above paddleOCR and pymupdf are free as well so it fits your project purpose.

@tkcoding commented on GitHub (Dec 13, 2024): @haniehm Sorry for the last reply , if you haven find the solution. Basically llama is still working on improving the image extraction ,specifically for images that looks like a bunch of text in an images often confuse the outcome of the extraction (extracted as bunch of text with delimiter). Solution: Hybrid with other framework like paddleOCR (best solution) or pymupdf (OK solution). To extract the images. Create a multi-vector retriever with images processed into text and taking markdown from llamaparse. This will yield the best quality so far. Note: Above paddleOCR and pymupdf are free as well so it fits your project purpose.
yindo commented 2026-02-16 00:17:38 -05:00
Author
Owner
Copy Link
Copy Source

@BinaryBrain commented on GitHub (Dec 17, 2024):

@tkcoding There's no images in the document. It's texts representing some charts and diagrams. The OCR won't help.

@haniehm On the figure where there's a lot of T T T T T T T T T T T T T T T T T T T T T T T, the markdown from the response shows:

Figure 1
**Description:** The figure illustrates the spatial arrangement of tea and coconut in an intercropping system. It shows the layout with specific measurements indicated for the spacing between the plants. The letters represent different types of plants:
- C - Coconut
- T - Tea
- G - Gliricidia

I don't really know what you expect the output to be but it seems to fit your prompt.

Furthermore, you can use the Premium Mode for free. It just cost more credits. Here your example in Markdown:
TRISL_Guideline_02_2021e_Aug2021Intercropping-1.pdf.md

@BinaryBrain commented on GitHub (Dec 17, 2024): @tkcoding There's no images in the document. It's texts representing some charts and diagrams. The OCR won't help. @haniehm On the figure where there's a lot of ` T T T T T T T T T T T T T T T T T T T T T T T `, the markdown from the response shows: ```md Figure 1 **Description:** The figure illustrates the spatial arrangement of tea and coconut in an intercropping system. It shows the layout with specific measurements indicated for the spacing between the plants. The letters represent different types of plants: - C - Coconut - T - Tea - G - Gliricidia ``` I don't really know what you expect the output to be but it seems to fit your prompt. Furthermore, you can use the Premium Mode for free. It just cost more credits. Here your example in Markdown: [TRISL_Guideline_02_2021e_Aug2021Intercropping-1.pdf.md](https://github.com/user-attachments/files/18170042/TRISL_Guideline_02_2021e_Aug2021Intercropping-1.pdf.md)
yindo commented 2026-02-16 00:17:39 -05:00
Author
Owner
Copy Link
Copy Source

@haniehm commented on GitHub (Dec 18, 2024):

@Thank you, @BinaryBrain and @tkcoding, for your assistance.

@BinaryBrain, I appreciate the clarification regarding the performance in Figure 1 and for sharing the markdown file—it’s very helpful. My primary concern lies with Figure 2, as it wasn’t extracted correctly. That said, I believe these are edge cases, as LlamaParse has performed well on other documents.

@haniehm commented on GitHub (Dec 18, 2024): @Thank you, @BinaryBrain and @tkcoding, for your assistance. @BinaryBrain, I appreciate the clarification regarding the performance in Figure 1 and for sharing the markdown file—it’s very helpful. My primary concern lies with Figure 2, as it wasn’t extracted correctly. That said, I believe these are edge cases, as LlamaParse has performed well on other documents.
yindo commented 2026-02-16 00:17:39 -05:00
Author
Owner
Copy Link
Copy Source

@BinaryBrain commented on GitHub (Dec 20, 2024):

I'd like LlamaParse to be able to handle each and every edge cases.
Could you manually generate the kind of answer you want for Figure 2, so I can have a better understanding on what should be the output?

@BinaryBrain commented on GitHub (Dec 20, 2024): I'd like LlamaParse to be able to handle each and every edge cases. Could you manually generate the kind of answer you want for Figure 2, so I can have a better understanding on what should be the output?
yindo commented 2026-02-16 00:17:39 -05:00
Author
Owner
Copy Link
Copy Source

@haniehm commented on GitHub (Dec 20, 2024):

@BinaryBrain Sure this is the response I was hoping for Figure 2:
image

@haniehm commented on GitHub (Dec 20, 2024): @BinaryBrain Sure this is the response I was hoping for Figure 2: ![image](https://github.com/user-attachments/assets/34fb5a1e-4069-4b55-b5de-df3ce4ff3806)
yindo commented 2026-02-16 00:17:39 -05:00
Author
Owner
Copy Link
Copy Source

@BinaryBrain commented on GitHub (Dec 20, 2024):

@BinaryBrain Sure this is the response I was hoping for Figure 2

But how would you translate this in Markdown?

@BinaryBrain commented on GitHub (Dec 20, 2024): > @BinaryBrain Sure this is the response I was hoping for Figure 2 But how would you translate this in Markdown?
yindo commented 2026-02-16 00:17:39 -05:00
Author
Owner
Copy Link
Copy Source

@haniehm commented on GitHub (Dec 20, 2024):

@BinaryBrain this is Figure 1 in the same document and I was hoping to get image below:
image
Interesting that llamaparse did detect the below Figure, which is in another document and has some similarities to figure 1 in the sense of using letters, very well with the prompt shared before .
image

@haniehm commented on GitHub (Dec 20, 2024): @BinaryBrain this is Figure 1 in the same document and I was hoping to get image below: ![image](https://github.com/user-attachments/assets/b097180f-48e1-4a25-a943-ecf1cc117d2c) Interesting that llamaparse did detect the below Figure, which is in another document and has some similarities to figure 1 in the sense of using letters, very well with the prompt shared before . ![image](https://github.com/user-attachments/assets/ef515f05-2e3a-444d-a55a-1a4b490661f1)
yindo commented 2026-02-16 00:17:39 -05:00
Author
Owner
Copy Link
Copy Source

@BinaryBrain commented on GitHub (Dec 20, 2024):

I see. Out of the box, we don't detect it as an image (because it's not really an image) but we just released a layout detection option: https://docs.cloud.llamaindex.ai/llamaparse/features/layout_extraction
It's 1 extra credit per page but it may solve your problem.

@BinaryBrain commented on GitHub (Dec 20, 2024): I see. Out of the box, we don't detect it as an image (because it's not really an image) but we just released a layout detection option: https://docs.cloud.llamaindex.ai/llamaparse/features/layout_extraction It's 1 extra credit per page but it may solve your problem.
yindo commented 2026-02-16 00:17:39 -05:00
Author
Owner
Copy Link
Copy Source

@haniehm commented on GitHub (Dec 20, 2024):

I see. Out of the box, we don't detect it as an image (because it's not really an image) but we just released a layout detection option: https://docs.cloud.llamaindex.ai/llamaparse/features/layout_extraction It's 1 extra credit per page but it may solve your problem.

I am just wondering how the figure below was detected well and I got a .png file out of it using the prompt above but not for the other ones. This is a mystery to me.

image

@haniehm commented on GitHub (Dec 20, 2024): > I see. Out of the box, we don't detect it as an image (because it's not really an image) but we just released a layout detection option: https://docs.cloud.llamaindex.ai/llamaparse/features/layout_extraction It's 1 extra credit per page but it may solve your problem. I am just wondering how the figure below was detected well and I got a .png file out of it using the prompt above but not for the other ones. This is a mystery to me. ![image](https://github.com/user-attachments/assets/de9d6c74-95e7-4c53-b0bc-0abe88d07e43)
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
LlamaCloud
LlamaExtract
LlamaParse
bug
client
complex layout
documentation
duplicate
enhancement
figure
font
header footer
help wanted
ocr
parser
pdf_debug
pdf_debug
pull-request
Mirrored from GitHub Pull Request
 python-lib
question
server_issue
server_issue
ui
No Label enhancement
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: run-llama/llama_cloud_services#366
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?