Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #649] Large file size? #370

New Issue
Closed
opened 2026-02-22 18:19:10 -05:00 by yindo · 3 comments
yindo commented 2026-02-22 18:19:10 -05:00
Owner
Copy Link
Copy Source

Originally created by @xiaoxiao261258 on GitHub (Jan 25, 2024).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/649

What would you like to see?

I tried to upload a big PDF file which was about 330M for 3 times and they were all failed.

Originally created by @xiaoxiao261258 on GitHub (Jan 25, 2024). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/649 ### What would you like to see? I tried to upload a big PDF file which was about 330M for 3 times and they were all failed.
yindo added the possible bugneeds info / can't replicate labels 2026-02-22 18:19:11 -05:00
yindo closed this issue 2026-02-22 18:19:11 -05:00
yindo commented 2026-02-22 18:19:11 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jan 26, 2024):

Can you please explain your bug?

  • How are you running AnythingLLM?
  • What embedder are you using?
  • Do you see any logs available in Docker, if applicable - the full error will be in there.
@timothycarambat commented on GitHub (Jan 26, 2024): Can you please explain your bug? - How are you running AnythingLLM? - What embedder are you using? - Do you see any logs available in Docker, if applicable - the full error will be in there.
yindo commented 2026-02-22 18:19:11 -05:00
Author
Owner
Copy Link
Copy Source

@xiaoxiao261258 commented on GitHub (Jan 28, 2024):

Can you please explain your bug?

  • How are you running AnythingLLM?
  • What embedder are you using?
  • Do you see any logs available in Docker, if applicable - the full error will be in there.

I deployed with a docker as the guide told me.
I used a local AI model which was vicuna-13b-1.5.
When I tried to upload a mp4 file, the log was as follows.

[Conversion Required] .mp4 file detected - converting to .wav
[Conversion Processing]: 4KB converted
[Conversion Processing]: 37284KB converted
[Conversion Complete]: File converted to .wav!
Failed to load the native whisper model: TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11730:11)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at runNextTicks (node:internal/process/task_queues:64:3)
    at process.processImmediate (node:internal/timers:447:9)
    at async getModelFile (file:///app/collector/node_modules/@xenova/transformers/src/utils/hub.js:471:24)
    at async getModelJSON (file:///app/collector/node_modules/@xenova/transformers/src/utils/hub.js:575:18)
    at async Promise.all (index 0)
    at async loadTokenizer (file:///app/collector/node_modules/@xenova/transformers/src/tokenizers.js:52:16)
    at async AutoTokenizer.from_pretrained (file:///app/collector/node_modules/@xenova/transformers/src/tokenizers.js:3940:48)
    at async Promise.all (index 0) {
  cause: ConnectTimeoutError: Connect Timeout Error
      at onConnectTimeout (node:internal/deps/undici/undici:6869:28)
      at node:internal/deps/undici/undici:6825:50
      at Immediate._onImmediate (node:internal/deps/undici/undici:6857:13)
      at process.processImmediate (node:internal/timers:476:21) {
    code: 'UND_ERR_CONNECT_TIMEOUT'
  }
}
node:internal/process/promises:288
            triggerUncaughtException(err, true /* fromPromise */);
            ^

TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11730:11)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at runNextTicks (node:internal/process/task_queues:64:3)
    at process.processImmediate (node:internal/timers:447:9)
    at async getModelFile (file:///app/collector/node_modules/@xenova/transformers/src/utils/hub.js:471:24)
    at async getModelJSON (file:///app/collector/node_modules/@xenova/transformers/src/utils/hub.js:575:18)
    at async Promise.all (index 0)
    at async loadTokenizer (file:///app/collector/node_modules/@xenova/transformers/src/tokenizers.js:52:16)
    at async AutoTokenizer.from_pretrained (file:///app/collector/node_modules/@xenova/transformers/src/tokenizers.js:3940:48)
    at async Promise.all (index 0) {
  cause: ConnectTimeoutError: Connect Timeout Error
      at onConnectTimeout (node:internal/deps/undici/undici:6869:28)
      at node:internal/deps/undici/undici:6825:50
      at Immediate._onImmediate (node:internal/deps/undici/undici:6857:13)
      at process.processImmediate (node:internal/timers:476:21) {
    code: 'UND_ERR_CONNECT_TIMEOUT'
  }
}

When I tried to upload a PDF file, it said an error occured when process the file.Following was the log.

Warning: Indexing all PDF objects
Error
    at InvalidPDFExceptionClosure (/app/collector/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:452:35)
    at Object.<anonymous> (/app/collector/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:455:2)
    at __w_pdfjs_require__ (/app/collector/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:45:30)
    at Object.<anonymous> (/app/collector/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:7939:23)
    at __w_pdfjs_require__ (/app/collector/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:45:30)
    at /app/collector/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:88:18
    at /app/collector/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:91:10
    at webpackUniversalModuleDefinition (/app/collector/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:18:20)
    at Object.<anonymous> (/app/collector/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:25:3)
    at Module._compile (node:internal/modules/cjs/loader:1356:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1414:10)
    at Module.load (node:internal/modules/cjs/loader:1197:32)
    at Module._load (node:internal/modules/cjs/loader:1013:12)
    at ModuleWrap.<anonymous> (node:internal/modules/esm/translators:202:29)
    at ModuleJob.run (node:internal/modules/esm/module_job:195:25)
    at async ModuleLoader.import (node:internal/modules/esm/loader:336:24) {
  message: 'Invalid PDF structure'
}

For a docx file, there was also someting with wrong.

-- Working SPG-Maoxin Shen-User manual.docx --
Error: Can't find end of central directory : is this a zip file ? If it is, see https://stuk.github.io/jszip/documentation/howto/read_zip.html
    at ZipEntries.readEndOfCentral (/app/collector/node_modules/jszip/lib/zipEntries.js:167:23)
    at ZipEntries.load (/app/collector/node_modules/jszip/lib/zipEntries.js:255:14)
    at /app/collector/node_modules/jszip/lib/load.js:48:24
-- Working SPG-Maoxin Shen-User manual.docx --
Error: Can't find end of central directory : is this a zip file ? If it is, see https://stuk.github.io/jszip/documentation/howto/read_zip.html
    at ZipEntries.readEndOfCentral (/app/collector/node_modules/jszip/lib/zipEntries.js:167:23)
    at ZipEntries.load (/app/collector/node_modules/jszip/lib/zipEntries.js:255:14)
    at /app/collector/node_modules/jszip/lib/load.js:48:24

A pptx file was faild to upload too.

-- Working 2024 SRCX R&D E3 Management Guide.pptx --
Could not parse office or office-like file [OfficeParser]: Your file officeParserTemp/tempfiles/170625926973600000.pptx seems to be corrupted. If you are sure it is fine, please create a ticket in Issues on github with the file to reproduce error.
Resulting text content was empty for 2024 SRCX R&D E3 Management Guide.pptx.

At last, I tested it with a big .md file about 89M. It could be uploaded successfully but failed when embedding.

Document cc.md uploaded processed and successfully. It is now available in documents.
[TELEMETRY SENT] {
  event: 'document_uploaded',
  distinctId: '5a9be5db-2681-43cb-bfbb-8722eaa85ec4',
  properties: { runtime: 'docker' }
}
Adding new vectorized document into namespace chao1.chen
Chunks created from document: 100334
LocalAI:listModels Request failed with status code 500
addDocumentToNamespace LocalAI Failed to embed: [500]: Request failed with status code 500
Failed to vectorize cc.md

In fact, only .txt and .md files with small size were uploaded successfully.
I'm not sure if this problem is caused by the file being too large or not supported file formats.

@xiaoxiao261258 commented on GitHub (Jan 28, 2024): > Can you please explain your bug? > > * How are you running AnythingLLM? > * What embedder are you using? > * Do you see any logs available in Docker, if applicable - the full error will be in there. I deployed with a docker as the guide told me. I used a local AI model which was vicuna-13b-1.5. When I tried to upload a mp4 file, the log was as follows. ``` [Conversion Required] .mp4 file detected - converting to .wav [Conversion Processing]: 4KB converted [Conversion Processing]: 37284KB converted [Conversion Complete]: File converted to .wav! Failed to load the native whisper model: TypeError: fetch failed at Object.fetch (node:internal/deps/undici/undici:11730:11) at processTicksAndRejections (node:internal/process/task_queues:95:5) at runNextTicks (node:internal/process/task_queues:64:3) at process.processImmediate (node:internal/timers:447:9) at async getModelFile (file:///app/collector/node_modules/@xenova/transformers/src/utils/hub.js:471:24) at async getModelJSON (file:///app/collector/node_modules/@xenova/transformers/src/utils/hub.js:575:18) at async Promise.all (index 0) at async loadTokenizer (file:///app/collector/node_modules/@xenova/transformers/src/tokenizers.js:52:16) at async AutoTokenizer.from_pretrained (file:///app/collector/node_modules/@xenova/transformers/src/tokenizers.js:3940:48) at async Promise.all (index 0) { cause: ConnectTimeoutError: Connect Timeout Error at onConnectTimeout (node:internal/deps/undici/undici:6869:28) at node:internal/deps/undici/undici:6825:50 at Immediate._onImmediate (node:internal/deps/undici/undici:6857:13) at process.processImmediate (node:internal/timers:476:21) { code: 'UND_ERR_CONNECT_TIMEOUT' } } node:internal/process/promises:288 triggerUncaughtException(err, true /* fromPromise */); ^ TypeError: fetch failed at Object.fetch (node:internal/deps/undici/undici:11730:11) at processTicksAndRejections (node:internal/process/task_queues:95:5) at runNextTicks (node:internal/process/task_queues:64:3) at process.processImmediate (node:internal/timers:447:9) at async getModelFile (file:///app/collector/node_modules/@xenova/transformers/src/utils/hub.js:471:24) at async getModelJSON (file:///app/collector/node_modules/@xenova/transformers/src/utils/hub.js:575:18) at async Promise.all (index 0) at async loadTokenizer (file:///app/collector/node_modules/@xenova/transformers/src/tokenizers.js:52:16) at async AutoTokenizer.from_pretrained (file:///app/collector/node_modules/@xenova/transformers/src/tokenizers.js:3940:48) at async Promise.all (index 0) { cause: ConnectTimeoutError: Connect Timeout Error at onConnectTimeout (node:internal/deps/undici/undici:6869:28) at node:internal/deps/undici/undici:6825:50 at Immediate._onImmediate (node:internal/deps/undici/undici:6857:13) at process.processImmediate (node:internal/timers:476:21) { code: 'UND_ERR_CONNECT_TIMEOUT' } } ``` When I tried to upload a PDF file, it said an error occured when process the file.Following was the log. ``` Warning: Indexing all PDF objects Error at InvalidPDFExceptionClosure (/app/collector/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:452:35) at Object.<anonymous> (/app/collector/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:455:2) at __w_pdfjs_require__ (/app/collector/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:45:30) at Object.<anonymous> (/app/collector/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:7939:23) at __w_pdfjs_require__ (/app/collector/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:45:30) at /app/collector/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:88:18 at /app/collector/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:91:10 at webpackUniversalModuleDefinition (/app/collector/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:18:20) at Object.<anonymous> (/app/collector/node_modules/pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js:25:3) at Module._compile (node:internal/modules/cjs/loader:1356:14) at Module._extensions..js (node:internal/modules/cjs/loader:1414:10) at Module.load (node:internal/modules/cjs/loader:1197:32) at Module._load (node:internal/modules/cjs/loader:1013:12) at ModuleWrap.<anonymous> (node:internal/modules/esm/translators:202:29) at ModuleJob.run (node:internal/modules/esm/module_job:195:25) at async ModuleLoader.import (node:internal/modules/esm/loader:336:24) { message: 'Invalid PDF structure' } ``` For a docx file, there was also someting with wrong. ``` -- Working SPG-Maoxin Shen-User manual.docx -- Error: Can't find end of central directory : is this a zip file ? If it is, see https://stuk.github.io/jszip/documentation/howto/read_zip.html at ZipEntries.readEndOfCentral (/app/collector/node_modules/jszip/lib/zipEntries.js:167:23) at ZipEntries.load (/app/collector/node_modules/jszip/lib/zipEntries.js:255:14) at /app/collector/node_modules/jszip/lib/load.js:48:24 -- Working SPG-Maoxin Shen-User manual.docx -- Error: Can't find end of central directory : is this a zip file ? If it is, see https://stuk.github.io/jszip/documentation/howto/read_zip.html at ZipEntries.readEndOfCentral (/app/collector/node_modules/jszip/lib/zipEntries.js:167:23) at ZipEntries.load (/app/collector/node_modules/jszip/lib/zipEntries.js:255:14) at /app/collector/node_modules/jszip/lib/load.js:48:24 ``` A pptx file was faild to upload too. ``` -- Working 2024 SRCX R&D E3 Management Guide.pptx -- Could not parse office or office-like file [OfficeParser]: Your file officeParserTemp/tempfiles/170625926973600000.pptx seems to be corrupted. If you are sure it is fine, please create a ticket in Issues on github with the file to reproduce error. Resulting text content was empty for 2024 SRCX R&D E3 Management Guide.pptx. ``` At last, I tested it with a big .md file about 89M. It could be uploaded successfully but failed when embedding. ``` Document cc.md uploaded processed and successfully. It is now available in documents. [TELEMETRY SENT] { event: 'document_uploaded', distinctId: '5a9be5db-2681-43cb-bfbb-8722eaa85ec4', properties: { runtime: 'docker' } } Adding new vectorized document into namespace chao1.chen Chunks created from document: 100334 LocalAI:listModels Request failed with status code 500 addDocumentToNamespace LocalAI Failed to embed: [500]: Request failed with status code 500 Failed to vectorize cc.md ``` In fact, only .txt and .md files with small size were uploaded successfully. I'm not sure if this problem is caused by the file being too large or not supported file formats.
yindo commented 2026-02-22 18:19:11 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jan 29, 2024):

Almost all of this seems to be from the collector.

  1. The mp4 file failed because your container could not download the hugging face model used to parse audio from mp4s. We have seen this happen to users using VPNs, in restricted IPs for using HuggingFace, or if running the docker container in an emulation mode.

  2. The other failures seem to do with correct or invalid document types. Are you sure these documents are valid? Can you open the docx in Word or in Google Docs? There should be no limitation on file size and instead it looks to have to do with the file itself.

There is an opportunity for the document to be corrupted during upload. If you are running docker as outlined in our documents could you open the storage/documents folder and see if your files are in there and are openable?

The last error seems to be an internal server error from LocalAI, not from AnythingLLM. This would imply your connection to your local instance is invalid. If LocalAI is returning a 500 error then something it wrong with your LocalAI instance as well - that would be outside of this repo and the logs in LocalAI will tell you what is wrong on that end.

@timothycarambat commented on GitHub (Jan 29, 2024): Almost all of this seems to be from the `collector`. 1. The mp4 file failed because your container could not download the hugging face model used to parse audio from mp4s. We have seen this happen to users using VPNs, in restricted IPs for using HuggingFace, or if running the docker container in an emulation mode. 2. The other failures seem to do with correct or invalid document types. Are you sure these documents are valid? Can you open the docx in Word or in Google Docs? There should be no limitation on file size and instead it looks to have to do with the file itself. There is an opportunity for the document to be corrupted during upload. If you are running docker as outlined in our documents could you open the `storage/documents` folder and see if your files are in there and are openable? The last error seems to be an internal server error from LocalAI, not from AnythingLLM. This would imply your connection to your local instance is invalid. If LocalAI is returning a 500 error then something it wrong with your LocalAI instance as well - that would be outside of this repo and the logs in LocalAI will tell you what is wrong on that end.
yindo changed title from Large file size? to [GH-ISSUE #649] Large file size? 2026-06-05 14:34:57 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label needs info / can't replicate possible bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#370
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?