run-llama/llama_cloud_services
1
0
Fork 0
mirror of https://github.com/run-llama/llama_cloud_services.git synced 2026-08-27 01:15:06 -04:00
Code Issues 352 Packages Projects Releases 10 Wiki Activity

Please add the option to include extra info in async batch processing #336

New Issue
Open
opened 2026-02-16 00:17:32 -05:00 by yindo · 2 comments
yindo commented 2026-02-16 00:17:32 -05:00
Owner
Copy Link
Copy Source

Originally created by @MemoonaTahira on GitHub (Nov 18, 2024).

# async batch
documents = await parser.aload_data(["./my_file1.pdf", "./my_file2.pdf"])

When I run this, I get parsed text chunks and no metadata whatsoever, unlike running SimpleDirectoryReader for PDFs.
I need to append filename and path to the parsed results.

However, using the extra info option seems to work only for a single file:

file_name = "my_file1.pdf"
extra_info = {"file_name": file_name}

with open(f"./{file_name}", "rb") as f:
    # must provide extra_info with file_name key with passing file object
    documents = parser.load_data(f, extra_info=extra_info)

E.g. I have 180 files that give me about 600 chunks, I have no way of identifying which chunk belongs to which file.

And async processing is fast, I don't wish to use the SimpleDirectoryReader because it is very slow.

Originally created by @MemoonaTahira on GitHub (Nov 18, 2024). ``` # async batch documents = await parser.aload_data(["./my_file1.pdf", "./my_file2.pdf"]) ``` When I run this, I get parsed text chunks and no metadata whatsoever, unlike running SimpleDirectoryReader for PDFs. I need to append filename and path to the parsed results. However, using the extra info option seems to work only for a single file: ``` file_name = "my_file1.pdf" extra_info = {"file_name": file_name} with open(f"./{file_name}", "rb") as f: # must provide extra_info with file_name key with passing file object documents = parser.load_data(f, extra_info=extra_info) ``` E.g. I have 180 files that give me about 600 chunks, I have no way of identifying which chunk belongs to which file. And async processing is fast, I don't wish to use the SimpleDirectoryReader because it is very slow.
yindo added the enhancement label 2026-02-16 00:17:32 -05:00
yindo commented 2026-02-16 00:17:32 -05:00
Author
Owner
Copy Link
Copy Source

@adreichert commented on GitHub (Apr 25, 2025):

Running into the same issue.

@adreichert commented on GitHub (Apr 25, 2025): Running into the same issue.
yindo commented 2026-02-16 00:17:32 -05:00
Author
Owner
Copy Link
Copy Source

@adreichert commented on GitHub (Apr 25, 2025):

@MemoonaTahira After running into the issue, I noticed new methods parse/aparse return JobResults, which retain the path. Please take a look at these function. These are in the 0.6.16.

lp = LlamaParse(
        ...
    )
res = lp.parse("foo.pdf")
res.file_name # ->  foo.pdf
@adreichert commented on GitHub (Apr 25, 2025): @MemoonaTahira After running into the issue, I noticed new methods [`parse/aparse`](https://github.com/run-llama/llama_cloud_services/blob/main/llama_cloud_services/parse/base.py#L1154-L1159) return `JobResult`s, which retain the path. Please take a look at these function. These are in the 0.6.16. ``` lp = LlamaParse( ... ) res = lp.parse("foo.pdf") res.file_name # -> foo.pdf ```
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
LlamaCloud
LlamaExtract
LlamaParse
bug
client
complex layout
documentation
duplicate
enhancement
figure
font
header footer
help wanted
ocr
parser
pdf_debug
pdf_debug
pull-request
Mirrored from GitHub Pull Request
 python-lib
question
server_issue
server_issue
ui
No Label enhancement
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: run-llama/llama_cloud_services#336
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?