can not extract the images from pdf pages #210

Closed
opened 2026-02-16 00:17:08 -05:00 by yindo · 9 comments
Owner

Originally created by @vanshaw2017 on GitHub (Jul 25, 2024).

in my case ,i wanna to extract all images from every page in my pdf file,and i used json mode (paser.get_json_result()). I didnt get images from the pdf page but the whole image of the pdf page instead everytime. Any advice for this situation?

what i got
what i got
what i wannt
what i want

Originally created by @vanshaw2017 on GitHub (Jul 25, 2024). in my case ,i wanna to extract all images from every page in my pdf file,and i used json mode (**paser.get_json_result()**). I didnt get images from the pdf page but the whole image of the pdf page instead everytime. Any advice for this situation? what i got ![what i got ](https://github.com/user-attachments/assets/83d157d7-54a4-43b5-b792-819ca0999b49) what i wannt ![what i want ](https://github.com/user-attachments/assets/c14a2639-415f-4e0b-aa16-08e17f611691)
yindo closed this issue 2026-02-16 00:17:08 -05:00
Author
Owner

@tkcoding commented on GitHub (Jul 25, 2024):

@vanshaw2017 Can you post with what's the code you use to extract?

I believe this is attention is all you need paper.
I tried it on my end and it works fine.

from llama_parse import LlamaParse
from llama_index.core import StorageContext
from llama_index.core import VectorStoreIndex
from llama_index.core.node_parser import MarkdownElementNodeParser
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding


from llama_index.core.schema import ImageDocument
from typing import List
from llama_index.core.node_parser import LlamaParseJsonNodeParser
from llama_index.core.schema import BaseNode, TextNode, Document
from llama_index.core.tools import QueryEngineTool, ToolMetadata
from llama_index.core.query_engine import SubQuestionQueryEngine
import glob
import nest_asyncio
import os
from dotenv import load_dotenv

load_dotenv()
nest_asyncio.apply()


ins = """
You are a highly proficient language model designed to convert pages from PDF, PPT and other files into structured markdown text. Your goal is to accurately transcribe text, represent formulas in LaTeX MathJax notation, and identify and describe images, particularly graphs and other graphical elements.

You have been tasked with creating a markdown copy of each page from the provided PDF or PPT image. Each image description must include a full description of the content, a summary of the graphical object.

Maintain the sequence of all the elements.

For the following element, follow the requirement of extraction:
for Text:
   - Extract all readable text from the page.
   - Exclude any diagonal text, headers, and footers.

for Text which includes hyperlink:
    -Extract hyperlink and present it with the text
    
for Formulas:
   - Identify and convert all formulas into LaTeX MathJax notation.

for Image Identification and Description:
   - Identify all images, graphs, and other graphical elements on the page.
   - If image contains wording that is hard to extract , flag it with <unidentifiable section> instead of parsing.
   - For each image, include a full description of the content in the alt text, followed by a brief summary of the graphical object.
   - If the image has a subtitle or caption, include it in the description.
   - If the image has a formula convert it into LaTeX MathJax notation.
   - If the image has a organisation chart , convert it into a hierachical understandable format.
   - for graph , extract the value in table form as markdown representation

    
# OUTPUT INSTRUCTIONS

- Ensure all formulas are in LaTeX MathJax notation.
- Exclude any diagonal text, headers, and footers from the output.
- For each image and graph, provide a detailed description and summary.
"""

class llama_document_parser(object):
    def __init__(self,parsing_ins):
        self.parser = LlamaParse(
            verbose=True,
            ignore_errors=False,
            invalidate_cache=True,
            do_not_cache=True,
        )

    def get_image_text_nodes(self,download_path: str,json_objs: List[dict]):
        """Extract out text from images using a multimodal model."""
        image_dicts = self.parser.get_images(json_objs, download_path=download_path)
        image_documents = []
        img_text_nodes = []
        for image_dict in image_dicts:
            image_doc = ImageDocument(image_path=image_dict["path"])
            img_text_nodes.append(image_doc)
        return img_text_nodes

    def document_processing_llamaparse(self,file_name: str ,image_output_folder:str):
        """Parse document in using llamaparse and return extracted elements in json format"""
        json_objs = self.parser.get_json_result(file_name)
        json_list = json_objs[0]["pages"]
        print(json_list)
        if not os.path.exists(image_output_folder):
            os.mkdir(image_output_folder)

        image_text_nodes = self.get_image_text_nodes(image_output_folder,json_objs)
        return json_list

llama_parser = llama_document_parser(parsing_ins=ins)
# llamaparse to extract documents
json_list = llama_parser.document_processing_llamaparse(file_name="docs/example_documents/attention_is_all_you_need.pdf",
                              image_output_folder="attention_is_allyouneed")

Output images:
image

@tkcoding commented on GitHub (Jul 25, 2024): @vanshaw2017 Can you post with what's the code you use to extract? I believe this is attention is all you need paper. I tried it on my end and it works fine. ``` from llama_parse import LlamaParse from llama_index.core import StorageContext from llama_index.core import VectorStoreIndex from llama_index.core.node_parser import MarkdownElementNodeParser from llama_index.llms.openai import OpenAI from llama_index.embeddings.openai import OpenAIEmbedding from llama_index.core.schema import ImageDocument from typing import List from llama_index.core.node_parser import LlamaParseJsonNodeParser from llama_index.core.schema import BaseNode, TextNode, Document from llama_index.core.tools import QueryEngineTool, ToolMetadata from llama_index.core.query_engine import SubQuestionQueryEngine import glob import nest_asyncio import os from dotenv import load_dotenv load_dotenv() nest_asyncio.apply() ins = """ You are a highly proficient language model designed to convert pages from PDF, PPT and other files into structured markdown text. Your goal is to accurately transcribe text, represent formulas in LaTeX MathJax notation, and identify and describe images, particularly graphs and other graphical elements. You have been tasked with creating a markdown copy of each page from the provided PDF or PPT image. Each image description must include a full description of the content, a summary of the graphical object. Maintain the sequence of all the elements. For the following element, follow the requirement of extraction: for Text: - Extract all readable text from the page. - Exclude any diagonal text, headers, and footers. for Text which includes hyperlink: -Extract hyperlink and present it with the text for Formulas: - Identify and convert all formulas into LaTeX MathJax notation. for Image Identification and Description: - Identify all images, graphs, and other graphical elements on the page. - If image contains wording that is hard to extract , flag it with <unidentifiable section> instead of parsing. - For each image, include a full description of the content in the alt text, followed by a brief summary of the graphical object. - If the image has a subtitle or caption, include it in the description. - If the image has a formula convert it into LaTeX MathJax notation. - If the image has a organisation chart , convert it into a hierachical understandable format. - for graph , extract the value in table form as markdown representation # OUTPUT INSTRUCTIONS - Ensure all formulas are in LaTeX MathJax notation. - Exclude any diagonal text, headers, and footers from the output. - For each image and graph, provide a detailed description and summary. """ class llama_document_parser(object): def __init__(self,parsing_ins): self.parser = LlamaParse( verbose=True, ignore_errors=False, invalidate_cache=True, do_not_cache=True, ) def get_image_text_nodes(self,download_path: str,json_objs: List[dict]): """Extract out text from images using a multimodal model.""" image_dicts = self.parser.get_images(json_objs, download_path=download_path) image_documents = [] img_text_nodes = [] for image_dict in image_dicts: image_doc = ImageDocument(image_path=image_dict["path"]) img_text_nodes.append(image_doc) return img_text_nodes def document_processing_llamaparse(self,file_name: str ,image_output_folder:str): """Parse document in using llamaparse and return extracted elements in json format""" json_objs = self.parser.get_json_result(file_name) json_list = json_objs[0]["pages"] print(json_list) if not os.path.exists(image_output_folder): os.mkdir(image_output_folder) image_text_nodes = self.get_image_text_nodes(image_output_folder,json_objs) return json_list llama_parser = llama_document_parser(parsing_ins=ins) # llamaparse to extract documents json_list = llama_parser.document_processing_llamaparse(file_name="docs/example_documents/attention_is_all_you_need.pdf", image_output_folder="attention_is_allyouneed") ``` Output images: ![image](https://github.com/user-attachments/assets/0e67d660-0d65-4ad8-a845-8aa011f7730b)
Author
Owner

@vanshaw2017 commented on GitHub (Jul 26, 2024):

Thanks a lot TK 😊,There is no big difference in the code in terms of api use .It is just the instruction prompt i used .I need to make more efforts on it . thanks!

@vanshaw2017 commented on GitHub (Jul 26, 2024): Thanks a lot TK 😊,There is no big difference in the code in terms of api use .It is just the instruction prompt i used .I need to make more efforts on it . thanks!
Author
Owner

@antonymott commented on GitHub (Jul 31, 2024):

@vanshaw2017 Although closed, I thought it may help to share that even when I followed the code of @tkcoding, depending on the resolution of the .pdf file being parsed, I also got whole page images saved in the images folder, not the individual figures/images (like your original issue). Some .pdf files are actually scans of older documents: text extraction works, but LlamaParse may not yet reliably differentiate scanned body text above or below an image as distinct from the image, so copies the entire page.

@antonymott commented on GitHub (Jul 31, 2024): @vanshaw2017 Although closed, I thought it may help to share that even when I followed the code of @tkcoding, depending on the resolution of the .pdf file being parsed, I also got whole page images saved in the images folder, not the individual figures/images (like your original issue). Some .pdf files are actually scans of older documents: text extraction works, but LlamaParse may not yet reliably differentiate scanned body text above or below an image as distinct from the image, so copies the entire page.
Author
Owner

@tkcoding commented on GitHub (Jul 31, 2024):

@antonymott
I agree with what you mentioned above .
I did a thorough research on parsing multiple elements document, there's no one tools that able to extract everything perfectly when comes to different variation of PDF that we could have.

I would suggest two steps approach instead of just dumping into any LLM api and hope the best.
First can go for offline extraction for images/table like PaddleOCR (this is by far the most comprehensive figure extraction) and second is send to GPT4o , llama3.1 or LLava to verify the content

With GPT4o-mini out, the cost is way more affordable now.

@tkcoding commented on GitHub (Jul 31, 2024): @antonymott I agree with what you mentioned above . I did a thorough research on parsing multiple elements document, there's no one tools that able to extract everything perfectly when comes to different variation of PDF that we could have. I would suggest two steps approach instead of just dumping into any LLM api and hope the best. First can go for offline extraction for images/table like PaddleOCR (this is by far the most comprehensive figure extraction) and second is send to GPT4o , llama3.1 or LLava to verify the content With GPT4o-mini out, the cost is way more affordable now.
Author
Owner

@antonymott commented on GitHub (Aug 1, 2024):

@tkcoding
yeah! Thanks for sharing PaddleOCR I hadn't heard of it. Have you tried sonnet3.5, often as amazing as 4o, I love them both. And yeah, cost is like $0.05 for insane access to cloud GPUs, often TPUs. Now with embeddings and vector-store functionality standard offerings from llms, we can build our own RAG pipelines for next to nothing ...unheard of just a year ago. It's actually incredibly exciting times...a whole new world of opportunity, even with a service as awesome as LlamaParse, we're by no means a slam-dunk with any one technology. I'm running 2 other LLMs, along with LlamaParse, with 3 text extraction python libraries, and use the non-LlamaParse LLMs initially just to evaluate and score the quality of each text extraction, which varies depending on the condition of the pdf. The python libraries also fail to differentiate the figures from grainy pages of text, which makes me think LlamaParse my be using them also. Thanks again for the tip about PaddleOCR...I'm happy to keep this closed comment thread going if you/@vanshaw2017 (or anyone else) is interested, and will share progress/improvements.

@antonymott commented on GitHub (Aug 1, 2024): @tkcoding yeah! Thanks for sharing PaddleOCR I hadn't heard of it. Have you tried sonnet3.5, often as amazing as 4o, I love them both. And yeah, cost is like $0.05 for insane access to cloud GPUs, often TPUs. Now with embeddings and vector-store functionality standard offerings from llms, we can build our own RAG pipelines for next to nothing ...unheard of just a year ago. It's actually incredibly exciting times...a whole new world of opportunity, even with a service as awesome as LlamaParse, we're by no means a slam-dunk with any one technology. I'm running 2 other LLMs, along with LlamaParse, with 3 text extraction python libraries, and use the non-LlamaParse LLMs initially just to evaluate and score the quality of each text extraction, which varies depending on the condition of the pdf. The python libraries also fail to differentiate the figures from grainy pages of text, which makes me think LlamaParse my be using them also. Thanks again for the tip about PaddleOCR...I'm happy to keep this closed comment thread going if you/@vanshaw2017 (or anyone else) is interested, and will share progress/improvements.
Author
Owner

@Doffy77k commented on GitHub (Aug 23, 2024):

@tkcoding
Are the extracted images from the pdf being converted to markdown file format?

@Doffy77k commented on GitHub (Aug 23, 2024): @tkcoding Are the extracted images from the pdf being converted to markdown file format?
Author
Owner

@fkemeth commented on GitHub (Oct 16, 2024):

Below is a leaner version of @tkcoding's solution. For me it works fine without any instructions.

I also created a blog post on this if you are interested, see here.

import os
import nest_asyncio
from llama_parse import LlamaParse
from llama_index.core.schema import ImageDocument, TextNode
from typing import List

nest_asyncio.apply()

FILE_NAME = "path to the article"
IMAGES_DOWNLOAD_PATH = "path to the folder where to store images"

LLAMA_CLOUD_API_KEY = os.environ["LLAMA_CLOUD_API_KEY"] # or your key

parser = LlamaParse(
    api_key=LLAMA_CLOUD_API_KEY,
    result_type="markdown",
)

json_objs = parser.get_json_result(FILE_NAME)
json_list = json_objs[0]["pages"]

def get_text_nodes(json_list: List[dict]) -> List[TextNode]:
    return [TextNode(text=page["text"], metadata={"page": page["page"]}) for page in json_list]

text_nodes = get_text_nodes(json_list)

def get_image_nodes(json_objs: List[dict], download_path: str) -> List[ImageDocument]:
    image_dicts = parser.get_images(json_objs, download_path=download_path)
    return [ImageDocument(image_path=image_dict["path"]) for image_dict in image_dicts]

image_documents = get_image_nodes(json_objs, IMAGES_DOWNLOAD_PATH)

Results:

grafik
@fkemeth commented on GitHub (Oct 16, 2024): Below is a leaner version of @tkcoding's solution. For me it works fine without any instructions. I also created a blog post on this if you are interested, see [here.](https://felixkemeth.medium.com/using-llamaparse-and-multimodal-llms-for-extracting-and-interpreting-text-and-images-from-pdfs-d201093b0e19) ``` import os import nest_asyncio from llama_parse import LlamaParse from llama_index.core.schema import ImageDocument, TextNode from typing import List nest_asyncio.apply() FILE_NAME = "path to the article" IMAGES_DOWNLOAD_PATH = "path to the folder where to store images" LLAMA_CLOUD_API_KEY = os.environ["LLAMA_CLOUD_API_KEY"] # or your key parser = LlamaParse( api_key=LLAMA_CLOUD_API_KEY, result_type="markdown", ) json_objs = parser.get_json_result(FILE_NAME) json_list = json_objs[0]["pages"] def get_text_nodes(json_list: List[dict]) -> List[TextNode]: return [TextNode(text=page["text"], metadata={"page": page["page"]}) for page in json_list] text_nodes = get_text_nodes(json_list) def get_image_nodes(json_objs: List[dict], download_path: str) -> List[ImageDocument]: image_dicts = parser.get_images(json_objs, download_path=download_path) return [ImageDocument(image_path=image_dict["path"]) for image_dict in image_dicts] image_documents = get_image_nodes(json_objs, IMAGES_DOWNLOAD_PATH) ``` Results: <img width="345" alt="grafik" src="https://github.com/user-attachments/assets/63bb8248-bf86-478b-a386-79a1a1c8f450">
Author
Owner

@jash0803 commented on GitHub (Dec 13, 2024):

@tkcoding I also want description of the image in the json i get. Is that possible?

@jash0803 commented on GitHub (Dec 13, 2024): @tkcoding I also want description of the image in the json i get. Is that possible?
Author
Owner

@tkcoding commented on GitHub (Dec 13, 2024):

@tkcoding I also want description of the image in the json i get. Is that possible?

In the markdown image will have the description describing the image (most probably you need to specify in prompt what image you would like to translate into text that will be helpful as well).

You can get the JSON object and try to look into the image object. I dont remember they give description in that section but you can try your luck.

@tkcoding commented on GitHub (Dec 13, 2024): > @tkcoding I also want description of the image in the json i get. Is that possible? In the markdown image will have the description describing the image (most probably you need to specify in prompt what image you would like to translate into text that will be helpful as well). You can get the JSON object and try to look into the image object. I dont remember they give description in that section but you can try your luck.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: run-llama/llama_cloud_services#210