run-llama/llama_cloud_services
1
0
Fork 0
mirror of https://github.com/run-llama/llama_cloud_services.git synced 2026-08-27 01:15:06 -04:00
Code Issues 352 Packages Projects Releases 10 Wiki Activity

gpt4o_mode=True parse output is showing unrelated context #188

New Issue
Open
opened 2026-02-16 00:17:05 -05:00 by yindo · 0 comments
yindo commented 2026-02-16 00:17:05 -05:00
Owner
Copy Link
Copy Source

Originally created by @tkcoding on GitHub (Jul 16, 2024).

**Issues : **
turned gpt4o_mode=True and return results is out of context. I parsed in a book that shows the forewords of author regarding to math and science however what I got in return with gpt4o_mode=True is about federal budget.

code:

from llama_parse import LlamaParse
from llama_index.core import StorageContext
from llama_index.core import VectorStoreIndex
from llama_index.core.node_parser import MarkdownElementNodeParser
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding
import nest_asyncio
import os
from dotenv import load_dotenv

load_dotenv()
nest_asyncio.apply()


import glob
from llama_parse import LlamaParse
from llama_index.core.schema import ImageDocument
from typing import List
from llama_index.core.node_parser import LlamaParseJsonNodeParser
from llama_index.llms.openai import OpenAI
from llama_index.core import VectorStoreIndex
from llama_index.core.schema import BaseNode, TextNode, Document
from llama_index.core.tools import QueryEngineTool, ToolMetadata
from llama_index.core.query_engine import SubQuestionQueryEngine

ins = """
You are a highly proficient language model designed to convert pages from PDF, PPT and other files into structured markdown text. Your goal is to accurately transcribe text, represent formulas in LaTeX MathJax notation, and identify and describe images, particularly graphs and other graphical elements.

You have been tasked with creating a markdown copy of each page from the provided PDF or PPT image. Each image description must include a full description of the content, a summary of the graphical object.

Maintain the sequence of all the elements.

For the following element, follow the requirement of extraction:
for Text:
   - Extract all readable text from the page.
   - Exclude any diagonal text, headers, and footers.

for Formulas:
   - Identify and convert all formulas into LaTeX MathJax notation.

for Image Identification and Description:
   - Identify all images, graphs, and other graphical elements on the page.
   - If image contains wording that is hard to extract , flag it with <unidentifiable section> instead of parsing.
   - For each image, include a full description of the content in the alt text, followed by a brief summary of the graphical object.
   - If the image has a subtitle or caption, include it in the description.
   - If the image has a formula convert it into LaTeX MathJax notation.
   - If the image has a organisation chart , convert it into a hierachical understandable format.
   - for graph , extract the value in table form as markdown representation

for Table:
    - If table span across multiple pages , make it into a single table in markdown.
    
# OUTPUT INSTRUCTIONS

- Ensure all formulas are in LaTeX MathJax notation.
- Exclude any diagonal text, headers, and footers from the output.
- For each image and graph, provide a detailed description and summary.
"""

class llama_document_parser(object):
    def __init__(self,parsing_ins):
        self.parser = LlamaParse(parsing_instruction=parsing_ins,language="en",gpt4o_mode=True,invalidate_cache=True,verbose=True)

    def get_image_text_nodes(self,download_path: str,json_objs: List[dict]):
        """Extract out text from images using a multimodal model."""
        image_dicts = self.parser.get_images(json_objs, download_path=download_path)
        image_documents = []
        img_text_nodes = []
        for image_dict in image_dicts:
            image_doc = ImageDocument(image_path=image_dict["path"])
            img_text_nodes.append(image_doc)
        return img_text_nodes

    def document_processing_llamaparse(self,file_name: str ,image_output_folder:str):
        """Parse document in using llamaparse and return extracted elements in json format"""
        json_objs = self.parser.get_json_result(file_name)
        json_list = json_objs[0]["pages"]
        print(json_list)
        if not os.path.exists(image_output_folder):
            os.mkdir(image_output_folder)

        image_text_nodes = self.get_image_text_nodes(image_output_folder,json_objs)
        return json_list


llama_parser = llama_document_parser(parsing_ins=ins)
json_list = llama_parser.document_processing_llamaparse(file_name="docs/example_documents/charles-5pages.pdf",
                              image_output_folder="charles")

File :
charles-5pages.pdf

Output:

Started parsing the file under job_id c96d3c5f-158d-44a1-aecb-77930dc99cb7
[{'page': 1, 'md': '# Analysis of 2000-2001 Federal Budget\n\n## Overview\n\nThe 2000-2001 Federal Budget was announced on 9 May 2000. This analysis provides an overview of the key features of the budget, including the economic context, major spending initiatives, and revenue measures.\n\n## Economic Context\n\n### Economic Growth\n- The Australian economy is expected to grow by 4% in 2000-2001.\n- Growth is supported by strong domestic demand, particularly in the housing and business investment sectors.\n\n### Inflation\n- Inflation is forecast to remain low, at around 2.5% in 2000-2001.\n\n### Employment\n- Employment growth is expected to be strong, with the unemployment rate projected to fall to 6.25% by June 2001.\n\n## Major Spending Initiatives\n\n### Health\n- $1.5 billion over four years for the Health and Aged Care portfolio.\n- Major initiatives include increased funding for public hospitals and aged care services.\n\n### Education\n- $1.2 billion over four years for the Education, Training and Youth Affairs portfolio.\n- Major initiatives include increased funding for schools and higher education institutions.\n\n### Defence\n- $500 million over four years for the Defence portfolio.\n- Major initiatives include increased funding for defence equipment and personnel.\n\n## Revenue Measures\n\n### Taxation\n- The Government will introduce a new tax system from 1 July 2000, including the Goods and Services Tax (GST).\n- The GST is expected to raise $24 billion in 2000-2001.\n\n### Other Revenue Measures\n- The Government will also introduce measures to improve tax compliance and reduce tax avoidance.\n\n## Conclusion\n\nThe 2000-2001 Federal Budget aims to support economic growth, maintain low inflation, and reduce unemployment. The budget includes significant spending initiatives in health, education, and defence, as well as major revenue measures, including the introduction of the GST.', 'images': [{'name': 'page-0.jpg', 'height': 0, 'width': 0, 'x': 0, 'y': 0}], 'items': [{'type': 'heading', 'lvl': 1, 'value': 'Analysis of 2000-2001 Federal Budget', 'md': '# Analysis of 2000-2001 Federal Budget'}, {'type': 'heading', 'lvl': 2, 'value': 'Overview', 'md': '## Overview'}, {'type': 'text', 'value': 'The 2000-2001 Federal Budget was announced on 9 May 2000. This analysis provides an overview of the key features of the budget, including the economic context, major spending initiatives, and revenue measures.', 'md': 'The 2000-2001 Federal Budget was announced on 9 May 2000. This analysis provides an overview of the key features of the budget, including the economic context, major spending initiatives, and revenue measures.'}, {'type': 'heading', 'lvl': 2, 'value': 'Economic Context', 'md': '## Economic Context'}, {'type': 'heading', 'lvl': 3, 'value': 'Economic Growth', 'md': '### Economic Growth'}, {'type': 'text', 'value': '- The Australian economy is expected to grow by 4% in 2000-2001.\n- Growth is supported by strong domestic demand, particularly in the housing and business investment sectors.', 'md': '- The Australian economy is expected to grow by 4% in 2000-2001.\n- Growth is supported by strong domestic demand, particularly in the housing and business investment sectors.'}, {'type': 'heading', 'lvl': 3, 'value': 'Inflation', 'md': '### Inflation'},

Originally created by @tkcoding on GitHub (Jul 16, 2024). **Issues : ** turned gpt4o_mode=True and return results is out of context. I parsed in a book that shows the forewords of author regarding to math and science however what I got in return with gpt4o_mode=True is about federal budget. **code:** ``` from llama_parse import LlamaParse from llama_index.core import StorageContext from llama_index.core import VectorStoreIndex from llama_index.core.node_parser import MarkdownElementNodeParser from llama_index.llms.openai import OpenAI from llama_index.embeddings.openai import OpenAIEmbedding import nest_asyncio import os from dotenv import load_dotenv load_dotenv() nest_asyncio.apply() import glob from llama_parse import LlamaParse from llama_index.core.schema import ImageDocument from typing import List from llama_index.core.node_parser import LlamaParseJsonNodeParser from llama_index.llms.openai import OpenAI from llama_index.core import VectorStoreIndex from llama_index.core.schema import BaseNode, TextNode, Document from llama_index.core.tools import QueryEngineTool, ToolMetadata from llama_index.core.query_engine import SubQuestionQueryEngine ins = """ You are a highly proficient language model designed to convert pages from PDF, PPT and other files into structured markdown text. Your goal is to accurately transcribe text, represent formulas in LaTeX MathJax notation, and identify and describe images, particularly graphs and other graphical elements. You have been tasked with creating a markdown copy of each page from the provided PDF or PPT image. Each image description must include a full description of the content, a summary of the graphical object. Maintain the sequence of all the elements. For the following element, follow the requirement of extraction: for Text: - Extract all readable text from the page. - Exclude any diagonal text, headers, and footers. for Formulas: - Identify and convert all formulas into LaTeX MathJax notation. for Image Identification and Description: - Identify all images, graphs, and other graphical elements on the page. - If image contains wording that is hard to extract , flag it with <unidentifiable section> instead of parsing. - For each image, include a full description of the content in the alt text, followed by a brief summary of the graphical object. - If the image has a subtitle or caption, include it in the description. - If the image has a formula convert it into LaTeX MathJax notation. - If the image has a organisation chart , convert it into a hierachical understandable format. - for graph , extract the value in table form as markdown representation for Table: - If table span across multiple pages , make it into a single table in markdown. # OUTPUT INSTRUCTIONS - Ensure all formulas are in LaTeX MathJax notation. - Exclude any diagonal text, headers, and footers from the output. - For each image and graph, provide a detailed description and summary. """ class llama_document_parser(object): def __init__(self,parsing_ins): self.parser = LlamaParse(parsing_instruction=parsing_ins,language="en",gpt4o_mode=True,invalidate_cache=True,verbose=True) def get_image_text_nodes(self,download_path: str,json_objs: List[dict]): """Extract out text from images using a multimodal model.""" image_dicts = self.parser.get_images(json_objs, download_path=download_path) image_documents = [] img_text_nodes = [] for image_dict in image_dicts: image_doc = ImageDocument(image_path=image_dict["path"]) img_text_nodes.append(image_doc) return img_text_nodes def document_processing_llamaparse(self,file_name: str ,image_output_folder:str): """Parse document in using llamaparse and return extracted elements in json format""" json_objs = self.parser.get_json_result(file_name) json_list = json_objs[0]["pages"] print(json_list) if not os.path.exists(image_output_folder): os.mkdir(image_output_folder) image_text_nodes = self.get_image_text_nodes(image_output_folder,json_objs) return json_list llama_parser = llama_document_parser(parsing_ins=ins) json_list = llama_parser.document_processing_llamaparse(file_name="docs/example_documents/charles-5pages.pdf", image_output_folder="charles") ``` **File :** [charles-5pages.pdf](https://github.com/user-attachments/files/16244870/charles-5pages.pdf) **Output:** Started parsing the file under job_id c96d3c5f-158d-44a1-aecb-77930dc99cb7 [{'page': 1, 'md': '# Analysis of 2000-2001 Federal Budget\n\n## Overview\n\nThe 2000-2001 Federal Budget was announced on 9 May 2000. This analysis provides an overview of the key features of the budget, including the economic context, major spending initiatives, and revenue measures.\n\n## Economic Context\n\n### Economic Growth\n- The Australian economy is expected to grow by 4% in 2000-2001.\n- Growth is supported by strong domestic demand, particularly in the housing and business investment sectors.\n\n### Inflation\n- Inflation is forecast to remain low, at around 2.5% in 2000-2001.\n\n### Employment\n- Employment growth is expected to be strong, with the unemployment rate projected to fall to 6.25% by June 2001.\n\n## Major Spending Initiatives\n\n### Health\n- $1.5 billion over four years for the Health and Aged Care portfolio.\n- Major initiatives include increased funding for public hospitals and aged care services.\n\n### Education\n- $1.2 billion over four years for the Education, Training and Youth Affairs portfolio.\n- Major initiatives include increased funding for schools and higher education institutions.\n\n### Defence\n- $500 million over four years for the Defence portfolio.\n- Major initiatives include increased funding for defence equipment and personnel.\n\n## Revenue Measures\n\n### Taxation\n- The Government will introduce a new tax system from 1 July 2000, including the Goods and Services Tax (GST).\n- The GST is expected to raise $24 billion in 2000-2001.\n\n### Other Revenue Measures\n- The Government will also introduce measures to improve tax compliance and reduce tax avoidance.\n\n## Conclusion\n\nThe 2000-2001 Federal Budget aims to support economic growth, maintain low inflation, and reduce unemployment. The budget includes significant spending initiatives in health, education, and defence, as well as major revenue measures, including the introduction of the GST.', 'images': [{'name': 'page-0.jpg', 'height': 0, 'width': 0, 'x': 0, 'y': 0}], 'items': [{'type': 'heading', 'lvl': 1, 'value': 'Analysis of 2000-2001 Federal Budget', 'md': '# Analysis of 2000-2001 Federal Budget'}, {'type': 'heading', 'lvl': 2, 'value': 'Overview', 'md': '## Overview'}, {'type': 'text', 'value': 'The 2000-2001 Federal Budget was announced on 9 May 2000. This analysis provides an overview of the key features of the budget, including the economic context, major spending initiatives, and revenue measures.', 'md': 'The 2000-2001 Federal Budget was announced on 9 May 2000. This analysis provides an overview of the key features of the budget, including the economic context, major spending initiatives, and revenue measures.'}, {'type': 'heading', 'lvl': 2, 'value': 'Economic Context', 'md': '## Economic Context'}, {'type': 'heading', 'lvl': 3, 'value': 'Economic Growth', 'md': '### Economic Growth'}, {'type': 'text', 'value': '- The Australian economy is expected to grow by 4% in 2000-2001.\n- Growth is supported by strong domestic demand, particularly in the housing and business investment sectors.', 'md': '- The Australian economy is expected to grow by 4% in 2000-2001.\n- Growth is supported by strong domestic demand, particularly in the housing and business investment sectors.'}, {'type': 'heading', 'lvl': 3, 'value': 'Inflation', 'md': '### Inflation'},
yindo added the bug label 2026-02-16 00:17:05 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
LlamaCloud
LlamaExtract
LlamaParse
bug
client
complex layout
documentation
duplicate
enhancement
figure
font
header footer
help wanted
ocr
parser
pdf_debug
pdf_debug
pull-request
Mirrored from GitHub Pull Request
 python-lib
question
server_issue
server_issue
ui
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: run-llama/llama_cloud_services#188
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?