Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-21 09:35:22 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #171] Can't process large pdf files (72 pages) #111

New Issue
Closed
opened 2026-02-22 18:17:56 -05:00 by yindo · 17 comments
yindo commented 2026-02-22 18:17:56 -05:00
Owner
Copy Link
Copy Source

Originally created by @nidupb on GitHub (Aug 2, 2023).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/171

When adding a 72 pdf pages (72 pages of table within a pdf), the console is giving that error :

2023-08-02 16:27:44 [2023-08-02 20:27:44 +0000] [13] [CRITICAL] WORKER TIMEOUT (pid:22)
2023-08-02 16:27:44 [2023-08-02 20:27:44 +0000] [22] [INFO] Worker exiting (pid: 22)
2023-08-02 16:27:44 Processing portfolio.pdf
2023-08-02 16:27:44 fetch failed
2023-08-02 16:27:44 Python processing API was not able to process document portfolio.pdf. Reason: fetch failed
2023-08-02 16:27:45 [2023-08-02 20:27:45 +0000] [91] [INFO] Booting worker with pid: 91

The UI does not display any error message.

When trying to take the 5 first pages of the pdf, it works correctly. I'm working locally with chroma DB.

Originally created by @nidupb on GitHub (Aug 2, 2023). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/171 When adding a 72 pdf pages (72 pages of table within a pdf), the console is giving that error : ``` 2023-08-02 16:27:44 [2023-08-02 20:27:44 +0000] [13] [CRITICAL] WORKER TIMEOUT (pid:22) 2023-08-02 16:27:44 [2023-08-02 20:27:44 +0000] [22] [INFO] Worker exiting (pid: 22) 2023-08-02 16:27:44 Processing portfolio.pdf 2023-08-02 16:27:44 fetch failed 2023-08-02 16:27:44 Python processing API was not able to process document portfolio.pdf. Reason: fetch failed 2023-08-02 16:27:45 [2023-08-02 20:27:45 +0000] [91] [INFO] Booting worker with pid: 91 ``` The UI does not display any error message. When trying to take the 5 first pages of the pdf, it works correctly. I'm working locally with chroma DB.
yindo added the question label 2026-02-22 18:17:56 -05:00
yindo closed this issue 2026-02-22 18:17:56 -05:00
yindo commented 2026-02-22 18:17:56 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Aug 3, 2023):

Is this when clicking to embed the documents into the vector DB or during the upload of the 72 page PDF into the document uploader?

@timothycarambat commented on GitHub (Aug 3, 2023): Is this when clicking to embed the documents into the vector DB or during the upload of the 72 page PDF into the document uploader?
yindo commented 2026-02-22 18:17:56 -05:00
Author
Owner
Copy Link
Copy Source

@nidupb commented on GitHub (Aug 3, 2023):

When uploading into the document uploader.

@nidupb commented on GitHub (Aug 3, 2023): When uploading into the document uploader.
yindo commented 2026-02-22 18:17:56 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Aug 3, 2023):

Is this document proprietary or is it public information? It would be nice to provide a download link to replicate the issue as I don't have access to a PDF quite that large. If not, I can try to produce one

@timothycarambat commented on GitHub (Aug 3, 2023): Is this document proprietary or is it public information? It would be nice to provide a download link to replicate the issue as I don't have access to a PDF quite that large. If not, I can try to produce one
yindo commented 2026-02-22 18:17:57 -05:00
Author
Owner
Copy Link
Copy Source

@nidupb commented on GitHub (Aug 3, 2023):

It's proprietary. Some indication though, it's an excel document converted to PDF, 10 columns * 800 rows, each cell contains mostly texts (kind of like of a business directory with some texts and essential informations for each of them). The pdf document is in landscape mode, standard letter format, and is 72 pages long.

Edit : After doing some tests, for some reason, I believe it's the conversion from office to a pdf files that makes it crash.

  • I tried to convert a gutenberg book (the 100 first pages) to pdf and the processing worked fine;
  • I copy paste those pages to a word document, then convert it to a pdf, and upload it in the document uploader, it failed.
@nidupb commented on GitHub (Aug 3, 2023): It's proprietary. Some indication though, it's an excel document converted to PDF, 10 columns * 800 rows, each cell contains mostly texts (kind of like of a business directory with some texts and essential informations for each of them). The pdf document is in landscape mode, standard letter format, and is 72 pages long. Edit : After doing some tests, for some reason, I believe it's the conversion from office to a pdf files that makes it crash. - I tried to convert a gutenberg book ([the 100 first pages](https://www.gutenberg.org/cache/epub/2600/pg2600-images.html)) to pdf and the processing worked fine; - I copy paste those pages to a word document, then convert it to a pdf, and upload it in the document uploader, it failed.
yindo commented 2026-02-22 18:17:57 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Aug 3, 2023):

But the export from word is perfectly readable in a PDF viewer?

@timothycarambat commented on GitHub (Aug 3, 2023): But the export from word is perfectly readable in a PDF viewer?
yindo commented 2026-02-22 18:17:57 -05:00
Author
Owner
Copy Link
Copy Source

@nidupb commented on GitHub (Aug 3, 2023):

yes it is, here is the file, since it's a public book.

@nidupb commented on GitHub (Aug 3, 2023): yes it is, [here is the file,](https://1drv.ms/b/s!AqICGC_gJjHziPkIOvu7M03gu7upkg?e=reFNtN) since it's a public book.
yindo commented 2026-02-22 18:17:57 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Aug 3, 2023):

I just downloaded that and uploaded locally and was able to then have the contents stage and available for embedding (did not embed to Pinecone though). The document collector was able to parse the pages correctly

@timothycarambat commented on GitHub (Aug 3, 2023): I just downloaded that and uploaded locally and was able to then have the contents stage and available for embedding (did not embed to Pinecone though). The document collector was able to parse the pages correctly
yindo commented 2026-02-22 18:17:57 -05:00
Author
Owner
Copy Link
Copy Source

@nidupb commented on GitHub (Aug 3, 2023):

That's weird, here is the log from docker on my side, since it's in docker, I wonder what in the environment make it failed :

2023-08-03 21:10:54 SELECT * FROM system_settings WHERE label = 'multi_user_mode'
2023-08-03 21:11:17 Processing guthenberg.pdf
2023-08-03 21:11:17 fetch failed
2023-08-03 21:11:17 Python processing API was not able to process document guthenberg.pdf. Reason: fetch failed
2023-08-03 21:11:17 [2023-08-04 01:11:17 +0000] [13] [CRITICAL] WORKER TIMEOUT (pid:5351)
2023-08-03 21:11:17 [2023-08-04 01:11:17 +0000] [5351] [INFO] Worker exiting (pid: 5351)
2023-08-03 21:11:17 [2023-08-04 01:11:17 +0000] [5878] [INFO] Booting worker with pid: 5878
@nidupb commented on GitHub (Aug 3, 2023): That's weird, here is the log from docker on my side, since it's in docker, I wonder what in the environment make it failed : ``` 2023-08-03 21:10:54 SELECT * FROM system_settings WHERE label = 'multi_user_mode' 2023-08-03 21:11:17 Processing guthenberg.pdf 2023-08-03 21:11:17 fetch failed 2023-08-03 21:11:17 Python processing API was not able to process document guthenberg.pdf. Reason: fetch failed 2023-08-03 21:11:17 [2023-08-04 01:11:17 +0000] [13] [CRITICAL] WORKER TIMEOUT (pid:5351) 2023-08-03 21:11:17 [2023-08-04 01:11:17 +0000] [5351] [INFO] Worker exiting (pid: 5351) 2023-08-03 21:11:17 [2023-08-04 01:11:17 +0000] [5878] [INFO] Booting worker with pid: 5878 ```
yindo commented 2026-02-22 18:17:57 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Aug 3, 2023):

If you visit http://localhost:8888/ while docker is running do you get a 200 response with Use POST /process with filename key in JSON body in order to process a file. File by that name must exist in hotdir already.

@timothycarambat commented on GitHub (Aug 3, 2023): If you visit `http://localhost:8888/` while docker is running do you get a 200 response with `Use POST /process with filename key in JSON body in order to process a file. File by that name must exist in hotdir already.`
yindo commented 2026-02-22 18:17:57 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Aug 3, 2023):

fetch failed to me seems like the worker didn't boot or something else is going awry - there is no fetch that PYPDFLoader should be calling that I know of. There is no stray file in your host machine anythingllm/collector/hotdir folder is there?

@timothycarambat commented on GitHub (Aug 3, 2023): `fetch failed` to me seems like the worker didn't boot or something else is going awry - there is no fetch that `PYPDFLoader` should be calling that I know of. There is no stray file in your host machine `anythingllm/collector/hotdir` folder is there?
yindo commented 2026-02-22 18:17:57 -05:00
Author
Owner
Copy Link
Copy Source

@nidupb commented on GitHub (Aug 3, 2023):

  • Visiting http://localhost:8888/ doesn't produce anything in docker (neither in the browser). Do I miss something here (considering that it does process shorter file) ?
  • anythingllm/collector/hotdir folder contains only the guthenberg pdf file I'm trying to process (the one I shared) (and the __HOTDIR__.md file)

Edit : here is the log for a successful shorter doc :

2023-08-03 22:09:27 SELECT * FROM system_settings WHERE label = 'multi_user_mode'
2023-08-03 22:09:27 SELECT * FROM system_settings WHERE label = 'multi_user_mode'
2023-08-03 22:09:27 SELECT * FROM workspaces WHERE slug = 'test'
2023-08-03 22:09:27 SELECT * FROM workspace_documents WHERE workspaceId = 9 
2023-08-03 22:09:27 SELECT * FROM system_settings WHERE label = 'multi_user_mode'
2023-08-03 22:09:48 SELECT * FROM system_settings WHERE label = 'multi_user_mode'
2023-08-03 22:09:48 SELECT * FROM system_settings WHERE label = 'multi_user_mode'
2023-08-03 22:09:48 SELECT * FROM workspaces WHERE slug = 'test'
2023-08-03 22:09:48 SELECT * FROM workspace_documents WHERE workspaceId = 9 
2023-08-03 22:09:48 SELECT * FROM system_settings WHERE label = 'multi_user_mode'
2023-08-03 22:09:50 SELECT * FROM system_settings WHERE label = 'multi_user_mode'
2023-08-03 22:09:50 SELECT * FROM system_settings WHERE label = 'multi_user_mode'
2023-08-03 22:09:50 SELECT * FROM workspaces WHERE slug = 'test'
2023-08-03 22:09:50 SELECT * FROM workspace_documents WHERE workspaceId = 9 
2023-08-03 22:09:57 Document portfolio p1-5.pdf uploaded processed and successfully. It is now available in documents.
2023-08-03 22:09:58 SELECT * FROM system_settings WHERE label = 'multi_user_mode'
2023-08-03 22:09:58 SELECT * FROM system_settings WHERE label = 'multi_user_mode'
2023-08-03 22:09:58 SELECT * FROM workspaces WHERE slug = 'test'
2023-08-03 22:09:58 SELECT * FROM workspace_documents WHERE workspaceId = 9
@nidupb commented on GitHub (Aug 3, 2023): - Visiting `http://localhost:8888/` doesn't produce anything in docker (neither in the browser). Do I miss something here (considering that it does process shorter file) ? - `anythingllm/collector/hotdir` folder contains only the guthenberg pdf file I'm trying to process (the one I shared) (and the `__HOTDIR__.md` file) Edit : here is the log for a successful shorter doc : ``` 2023-08-03 22:09:27 SELECT * FROM system_settings WHERE label = 'multi_user_mode' 2023-08-03 22:09:27 SELECT * FROM system_settings WHERE label = 'multi_user_mode' 2023-08-03 22:09:27 SELECT * FROM workspaces WHERE slug = 'test' 2023-08-03 22:09:27 SELECT * FROM workspace_documents WHERE workspaceId = 9 2023-08-03 22:09:27 SELECT * FROM system_settings WHERE label = 'multi_user_mode' 2023-08-03 22:09:48 SELECT * FROM system_settings WHERE label = 'multi_user_mode' 2023-08-03 22:09:48 SELECT * FROM system_settings WHERE label = 'multi_user_mode' 2023-08-03 22:09:48 SELECT * FROM workspaces WHERE slug = 'test' 2023-08-03 22:09:48 SELECT * FROM workspace_documents WHERE workspaceId = 9 2023-08-03 22:09:48 SELECT * FROM system_settings WHERE label = 'multi_user_mode' 2023-08-03 22:09:50 SELECT * FROM system_settings WHERE label = 'multi_user_mode' 2023-08-03 22:09:50 SELECT * FROM system_settings WHERE label = 'multi_user_mode' 2023-08-03 22:09:50 SELECT * FROM workspaces WHERE slug = 'test' 2023-08-03 22:09:50 SELECT * FROM workspace_documents WHERE workspaceId = 9 2023-08-03 22:09:57 Document portfolio p1-5.pdf uploaded processed and successfully. It is now available in documents. 2023-08-03 22:09:58 SELECT * FROM system_settings WHERE label = 'multi_user_mode' 2023-08-03 22:09:58 SELECT * FROM system_settings WHERE label = 'multi_user_mode' 2023-08-03 22:09:58 SELECT * FROM workspaces WHERE slug = 'test' 2023-08-03 22:09:58 SELECT * FROM workspace_documents WHERE workspaceId = 9 ```
yindo commented 2026-02-22 18:17:57 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Aug 3, 2023):

If you are processing documents then the document processor is online. In theory, if it was offline it would not allow you to upload at all.

Can you attempt the following:

  • Pull in latest branch
  • Delete any extra files in the host folder of your collector/hotdir
  • rebuild docker image on latest master commit
  • Try to upload guthenberg.pdf again and see if it is able to split that doc.

What OS are you running on? If Windows there could be a host of issues that are windows<>docker related but seeing that it is intermittent is odd and indicates a bug.

Also, what is the size in mb of the file you are uploading?

@timothycarambat commented on GitHub (Aug 3, 2023): If you are processing documents then the document processor is online. In theory, if it was offline it would not allow you to upload at all. Can you attempt the following: - Pull in latest branch - Delete any extra files in the host folder of your `collector/hotdir` - rebuild docker image on latest `master` commit - Try to upload [guthenberg.pdf](https://github.com/Mintplex-Labs/anything-llm/files/12256813/guthenberg.pdf) again and see if it is able to split that doc. What OS are you running on? If Windows there could be a host of issues that are windows<>docker related but seeing that it is intermittent is odd and indicates a bug. Also, what is the size in `mb` of the file you are uploading?
yindo commented 2026-02-22 18:17:57 -05:00
Author
Owner
Copy Link
Copy Source

@nidupb commented on GitHub (Aug 10, 2023):

So I did pull the last branch, rebuild the docker and uploaded again the guthenberg.pdf file with no success, same error.
I'm running on Mac OS Ventura 13.5. The guthenberg file is about 1.3 Mo

EDIT : I switched to pinecone, I'm having the same fetch error

2023-08-10 17:03:56 [2023-08-10 21:03:56 +0000] [13] [CRITICAL] WORKER TIMEOUT (pid:26)
2023-08-10 17:03:56 [2023-08-10 21:03:56 +0000] [26] [INFO] Worker exiting (pid: 26)
2023-08-10 17:03:56 Processing guthenberg.pdf
2023-08-10 17:03:56 fetch failed
2023-08-10 17:03:56 Python processing API was not able to process document guthenberg.pdf. Reason: fetch failed
2023-08-10 17:03:56 [2023-08-10 21:03:56 +0000] [91] [INFO] Booting worker with pid: 91
@nidupb commented on GitHub (Aug 10, 2023): So I did pull the last branch, rebuild the docker and uploaded again the guthenberg.pdf file with no success, same error. I'm running on Mac OS Ventura 13.5. The guthenberg file is about 1.3 Mo EDIT : I switched to pinecone, I'm having the same fetch error ``` 2023-08-10 17:03:56 [2023-08-10 21:03:56 +0000] [13] [CRITICAL] WORKER TIMEOUT (pid:26) 2023-08-10 17:03:56 [2023-08-10 21:03:56 +0000] [26] [INFO] Worker exiting (pid: 26) 2023-08-10 17:03:56 Processing guthenberg.pdf 2023-08-10 17:03:56 fetch failed 2023-08-10 17:03:56 Python processing API was not able to process document guthenberg.pdf. Reason: fetch failed 2023-08-10 17:03:56 [2023-08-10 21:03:56 +0000] [91] [INFO] Booting worker with pid: 91 ```
yindo commented 2026-02-22 18:17:57 -05:00
Author
Owner
Copy Link
Copy Source

@franzbischoff commented on GitHub (Aug 16, 2023):

This may be fixed on PR #183

@franzbischoff commented on GitHub (Aug 16, 2023): This may be fixed on PR #183
yindo commented 2026-02-22 18:17:57 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Aug 16, 2023):

Fixed by 1b4e29a3b9 where we bump timeout of document processer to 300 seconds. Default was 30 which for larger PDFs will not succeed.

@timothycarambat commented on GitHub (Aug 16, 2023): Fixed by 1b4e29a3b9a642bad4f5d1358fd568b3491674aa where we bump timeout of document processer to 300 seconds. Default was 30 which for larger PDFs will not succeed.
yindo commented 2026-02-22 18:17:57 -05:00
Author
Owner
Copy Link
Copy Source

@nidupb commented on GitHub (Aug 17, 2023):

Indeed it works, thanks. However I'd suggest to go higher than 300 seconds for very large document with some kind of way to indicate the processing.

@nidupb commented on GitHub (Aug 17, 2023): Indeed it works, thanks. However I'd suggest to go higher than 300 seconds for very large document with some kind of way to indicate the processing.
yindo commented 2026-02-22 18:17:57 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Aug 17, 2023):

Knowing it works i am okay going much higher now - I agree

@timothycarambat commented on GitHub (Aug 17, 2023): Knowing it works i am okay going much higher now - I agree
yindo changed title from Can't process large pdf files (72 pages) to [GH-ISSUE #171] Can't process large pdf files (72 pages) 2026-06-05 14:33:32 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label question
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#111
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?