run-llama/llama_cloud_services
1
0
Fork 0
mirror of https://github.com/run-llama/llama_cloud_services.git synced 2026-07-19 16:43:32 -04:00
Code Issues 352 Packages Projects Releases 10 Wiki Activity

[PR #254] [MERGED] Modify _get_sub_docs to use Custom Separator #694

New Issue
Closed
opened 2026-02-16 00:18:49 -05:00 by yindo · 0 comments
yindo commented 2026-02-16 00:18:49 -05:00
Owner
Copy Link
Copy Source

📋 Pull Request Information

Original PR: https://github.com/run-llama/llama_cloud_services/pull/254
Author: @adreichert
Created: 6/27/2024
Status: ✅ Merged
Merged: 7/17/2024
Merged by: @jerryjliu

Base: main ← Head: adreichert/split-on-custom-separator

📝 Commits (2)

  • b644065 Move _get_sub_docs to private function
  • 6dd9230 Merge branch 'main' into adreichert/split-on-custom-separator

📊 Changes

1 file changed (+18 additions, -16 deletions)

View changed files

📝 llama_parse/base.py (+18 -16)

📄 Description

Summary

This PR modifies _get_sub_docs to use the separator passed into the LlamaParse constructor. I'm making this change as the string \n---\n occurs occasionally in our documents. If pagination is important, we need to use a separator less likely to occur in our documents such as \n$$$$$$$$\n.

Testing

CLI

Automated Tests passed

% export LLAMA_CLOUD_API_KEY=llx-[...]
% make test                                                                      
pytest tests
====================================================================== test session starts =======================================================================
platform darwin -- Python 3.11.7, pytest-8.2.2, pluggy-1.5.0
rootdir: /Users/areichert/Documents/llama_parse
configfile: pyproject.toml
plugins: anyio-4.4.0
collected 3 items                                                                                                                                                

tests/test_reader.py ...                                                                                                                                   [100%]

======================================================================= 3 passed in 15.01s =======================================================================

Test Script

We parsed this two page document, which has a \n---\n where the background color changes.

[...]
**TO HELP ENGAGE EMPLOYEES**

---

Fulkrum has been providing inspection, [...]

Test Script

import llama_parse
LLAMAPARSE_API_KEY = '[...]'
OPENAI_API_KEY = '[...]'

def parse(split_by_page, page_separator):
    print(f"{split_by_page=}, {page_separator=}")
    parser = llama_parse.LlamaParse(
        result_type='markdown',
        api_key=LLAMAPARSE_API_KEY,
        verbose=False,
        invalidate_cache=True,
        gpt4o_mode=True,
        gpt4o_api_key=OPENAI_API_KEY,
        ignore_errors=True,
        split_by_page=split_by_page,
        page_separator=page_separator,
    )
    result = parser.load_data('fulkrum.pdf')
    print(f"{len(result)} pages")


if __name__ == '__main__':
    parse(False, None)
    parse(True, None)
    parse(True, "\n$$$$$$$$\n")

Results

  • When not splitting by page, the output list contains 1 element
  • When using the default separator, we get five, an incorrect number of pages
  • When using the separator that doesn't appear in the document, we get the correct number of pages.
% python test2.py
split_by_page=False, page_separator=None
1 pages
split_by_page=True, page_separator=None
5 pages
split_by_page=True, page_separator='\n$$$$$$$$\n'
2 pages

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/run-llama/llama_cloud_services/pull/254 **Author:** [@adreichert](https://github.com/adreichert) **Created:** 6/27/2024 **Status:** ✅ Merged **Merged:** 7/17/2024 **Merged by:** [@jerryjliu](https://github.com/jerryjliu) **Base:** `main` ← **Head:** `adreichert/split-on-custom-separator` --- ### 📝 Commits (2) - [`b644065`](https://github.com/run-llama/llama_cloud_services/commit/b644065c85175f75843b2764c324126d7fcf5310) Move _get_sub_docs to private function - [`6dd9230`](https://github.com/run-llama/llama_cloud_services/commit/6dd9230c02783366691be36078b4432d0594c840) Merge branch 'main' into adreichert/split-on-custom-separator ### 📊 Changes **1 file changed** (+18 additions, -16 deletions) <details> <summary>View changed files</summary> 📝 `llama_parse/base.py` (+18 -16) </details> ### 📄 Description # Summary This PR modifies ` _get_sub_docs` to use the separator passed into the `LlamaParse` constructor. I'm making this change as the string `\n---\n` occurs occasionally in our documents. If pagination is important, we need to use a separator less likely to occur in our documents such as `\n$$$$$$$$\n`. # Testing ## CLI Automated Tests passed ``` % export LLAMA_CLOUD_API_KEY=llx-[...] % make test pytest tests ====================================================================== test session starts ======================================================================= platform darwin -- Python 3.11.7, pytest-8.2.2, pluggy-1.5.0 rootdir: /Users/areichert/Documents/llama_parse configfile: pyproject.toml plugins: anyio-4.4.0 collected 3 items tests/test_reader.py ... [100%] ======================================================================= 3 passed in 15.01s ======================================================================= ``` # Test Script We parsed [this two page document](https://www.bamboohr.com/customers/assets/fulkrum.pdf), which has a `\n---\n` where the background color changes. ``` [...] **TO HELP ENGAGE EMPLOYEES** --- Fulkrum has been providing inspection, [...] ``` ## Test Script ``` import llama_parse LLAMAPARSE_API_KEY = '[...]' OPENAI_API_KEY = '[...]' def parse(split_by_page, page_separator): print(f"{split_by_page=}, {page_separator=}") parser = llama_parse.LlamaParse( result_type='markdown', api_key=LLAMAPARSE_API_KEY, verbose=False, invalidate_cache=True, gpt4o_mode=True, gpt4o_api_key=OPENAI_API_KEY, ignore_errors=True, split_by_page=split_by_page, page_separator=page_separator, ) result = parser.load_data('fulkrum.pdf') print(f"{len(result)} pages") if __name__ == '__main__': parse(False, None) parse(True, None) parse(True, "\n$$$$$$$$\n") ``` ## Results * When not splitting by page, the output list contains 1 element * When using the default separator, we get five, an incorrect number of pages * When using the separator that doesn't appear in the document, we get the correct number of pages. ``` % python test2.py split_by_page=False, page_separator=None 1 pages split_by_page=True, page_separator=None 5 pages split_by_page=True, page_separator='\n$$$$$$$$\n' 2 pages ``` --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
yindo added the pull-request label 2026-02-16 00:18:49 -05:00
yindo closed this issue 2026-02-16 00:18:50 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
LlamaCloud
LlamaExtract
LlamaParse
bug
client
complex layout
documentation
duplicate
enhancement
figure
font
header footer
help wanted
ocr
parser
pdf_debug
pdf_debug
pull-request
Mirrored from GitHub Pull Request
 python-lib
question
server_issue
server_issue
ui
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: run-llama/llama_cloud_services#694
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?