Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 22:23:50 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[PR #5252] [MERGED] fix(collector): infer file extension from Content-Type for URLs without explicit extensions #5371

New Issue
Closed
opened 2026-06-05 15:21:09 -04:00 by yindo · 0 comments
yindo commented 2026-06-05 15:21:09 -04:00
Owner
Copy Link
Copy Source

📋 Pull Request Information

Original PR: https://github.com/Mintplex-Labs/anything-llm/pull/5252
Author: @Lyt060814
Created: 3/22/2026
Status: ✅ Merged
Merged: 3/23/2026
Merged by: @timothycarambat

Base: master ← Head: fix/url-download-missing-extension

📝 Commits (2)

  • 5045daf fix(collector): infer file extension from Content-Type for URLs without explicit extensions
  • 74e9eb3 small refactor

📊 Changes

3 files changed (+151 additions, -7 deletions)

View changed files

➕ collector/__tests__/utils/downloadURIToFile/index.test.js (+96 -0)
📝 collector/processLink/helpers/index.js (+14 -2)
📝 collector/utils/downloadURIToFile/index.js (+41 -5)

📄 Description

Problem

When downloading files from URLs that lack explicit file extensions (e.g., https://arxiv.org/pdf/2307.10265), the collector fails to process them because the filename gets no recognized extension.

The error:

File extension .10265 not supported for parsing and cannot be assumed as text file type.

Root Cause

downloadURIToFile builds the local filename purely from the URL path via slugify. For URLs like the arxiv example, this produces a filename like arxiv.org-pdf-230710265 (or arxiv.org-pdf-2307.10265) — neither has a recognized extension. When processSingleFile later inspects the extension, it finds .10265 (or none), which isn't in SUPPORTED_FILETYPE_CONVERTERS, and rejects the file.

Fix

After downloading, check whether the saved filename has a recognized file extension. If not, inspect the response's Content-Type header and map it to the correct extension using the existing ACCEPTED_MIMES lookup table.

For example:

  • URL: https://arxiv.org/pdf/2307.10265
  • Response header: Content-Type: application/pdf
  • File is saved as arxiv.org-pdf-230710265.pdf → processed correctly ✅

This approach:

  • Reuses the existing ACCEPTED_MIMES map (no new dependencies)
  • Only activates when the URL path doesn't already have a supported extension (no behavior change for well-formed URLs)
  • Handles Content-Type with charset parameters (e.g., application/pdf; charset=utf-8)

Tests

Added 15 unit tests covering:

  • MIME-to-extension mapping for all supported types
  • The arxiv URL case (no extension)
  • URLs with numeric-looking false extensions (.10265)
  • URLs that already have correct extensions (no double-append)
  • Unknown/null content types (graceful no-op)
  • Content-Type with charset parameters

All existing tests continue to pass.

Fixes #4513

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/Mintplex-Labs/anything-llm/pull/5252 **Author:** [@Lyt060814](https://github.com/Lyt060814) **Created:** 3/22/2026 **Status:** ✅ Merged **Merged:** 3/23/2026 **Merged by:** [@timothycarambat](https://github.com/timothycarambat) **Base:** `master` ← **Head:** `fix/url-download-missing-extension` --- ### 📝 Commits (2) - [`5045daf`](https://github.com/Mintplex-Labs/anything-llm/commit/5045daf72a5af33a3456c13b7aeb8349b8a55268) fix(collector): infer file extension from Content-Type for URLs without explicit extensions - [`74e9eb3`](https://github.com/Mintplex-Labs/anything-llm/commit/74e9eb3764aeccf14327ee0d12e9e942c4a7dce5) small refactor ### 📊 Changes **3 files changed** (+151 additions, -7 deletions) <details> <summary>View changed files</summary> ➕ `collector/__tests__/utils/downloadURIToFile/index.test.js` (+96 -0) 📝 `collector/processLink/helpers/index.js` (+14 -2) 📝 `collector/utils/downloadURIToFile/index.js` (+41 -5) </details> ### 📄 Description ## Problem When downloading files from URLs that lack explicit file extensions (e.g., `https://arxiv.org/pdf/2307.10265`), the collector fails to process them because the filename gets no recognized extension. The error: ``` File extension .10265 not supported for parsing and cannot be assumed as text file type. ``` ## Root Cause `downloadURIToFile` builds the local filename purely from the URL path via `slugify`. For URLs like the arxiv example, this produces a filename like `arxiv.org-pdf-230710265` (or `arxiv.org-pdf-2307.10265`) — neither has a recognized extension. When `processSingleFile` later inspects the extension, it finds `.10265` (or none), which isn't in `SUPPORTED_FILETYPE_CONVERTERS`, and rejects the file. ## Fix After downloading, check whether the saved filename has a recognized file extension. If not, inspect the response's `Content-Type` header and map it to the correct extension using the existing `ACCEPTED_MIMES` lookup table. For example: - URL: `https://arxiv.org/pdf/2307.10265` - Response header: `Content-Type: application/pdf` - File is saved as `arxiv.org-pdf-230710265.pdf` → processed correctly ✅ This approach: - Reuses the existing `ACCEPTED_MIMES` map (no new dependencies) - Only activates when the URL path doesn't already have a supported extension (no behavior change for well-formed URLs) - Handles `Content-Type` with charset parameters (e.g., `application/pdf; charset=utf-8`) ## Tests Added 15 unit tests covering: - MIME-to-extension mapping for all supported types - The arxiv URL case (no extension) - URLs with numeric-looking false extensions (`.10265`) - URLs that already have correct extensions (no double-append) - Unknown/null content types (graceful no-op) - Content-Type with charset parameters All existing tests continue to pass. Fixes #4513 --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
yindo added the pull-request label 2026-06-05 15:21:09 -04:00
yindo closed this issue 2026-06-05 15:21:09 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#5371
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?