Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 22:23:50 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #3626] [FEAT]: Allow custom headers for the upload-link endpoint #2339

New Issue
Closed
opened 2026-02-22 18:29:14 -05:00 by yindo · 8 comments
yindo commented 2026-02-22 18:29:14 -05:00
Owner
Copy Link
Copy Source

Originally created by @agorman on GitHub (Apr 10, 2025).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/3626

Originally assigned to: @shatfield4 on GitHub.

What would you like to see?

When using the /v1/document/upload-link endpoint I'd like to be able to add my own custom headers. This would allow me to include API keys for links that require authorization.

Originally created by @agorman on GitHub (Apr 10, 2025). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/3626 Originally assigned to: @shatfield4 on GitHub. ### What would you like to see? When using the `/v1/document/upload-link` endpoint I'd like to be able to add my own custom headers. This would allow me to include API keys for links that require authorization.
yindo added the enhancementcore-team-onlyfeature request labels 2026-02-22 18:29:14 -05:00
yindo closed this issue 2026-02-22 18:29:14 -05:00
yindo commented 2026-02-22 18:29:15 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Apr 11, 2025):

How would we then pass these onto whatever the destination link would be? We run puppeteer on that part - so I presume we would append them before opening the page?

@timothycarambat commented on GitHub (Apr 11, 2025): How would we then pass these onto whatever the destination link would be? We run puppeteer on that part - so I presume we would append them before opening the page?
yindo commented 2026-02-22 18:29:15 -05:00
Author
Owner
Copy Link
Copy Source

@agorman commented on GitHub (Apr 11, 2025):

I'm not sure I follow. I'm running this in docker aren't you just using an http client in the backend to pass the call?

@agorman commented on GitHub (Apr 11, 2025): I'm not sure I follow. I'm running this in docker aren't you just using an http client in the backend to pass the call?
yindo commented 2026-02-22 18:29:15 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Apr 11, 2025):

No, we use pupteer and fallback to regular HTTP fetch if that fails, so that JS-enabled pages are rendered properly and can be collected once the content is fully rendered.
https://github.com/Mintplex-Labs/anything-llm/blob/master/collector/processLink/convert/generic.js#L72-L122

@timothycarambat commented on GitHub (Apr 11, 2025): No, we use pupteer and fallback to regular HTTP fetch if that fails, so that JS-enabled pages are rendered properly and can be collected once the content is fully rendered. https://github.com/Mintplex-Labs/anything-llm/blob/master/collector/processLink/convert/generic.js#L72-L122
yindo commented 2026-02-22 18:29:15 -05:00
Author
Owner
Copy Link
Copy Source

@agorman commented on GitHub (Apr 11, 2025):

Thank you for the explanation and the link. That makes sense now. Would something like this work? https://pptr.dev/api/puppeteer.page.setextrahttpheaders

@agorman commented on GitHub (Apr 11, 2025): Thank you for the explanation and the link. That makes sense now. Would something like this work? https://pptr.dev/api/puppeteer.page.setextrahttpheaders
yindo commented 2026-02-22 18:29:15 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Apr 14, 2025):

@agorman Absolutely, that would be the main idea here. However, I think passing auth could be tricky here since often that can be site-specific. So the issue becomes pretty wide in impact.

For example, consider basic JWT Authentication Bearer headers. That same header is used across many sites, but the token changes per site. So there would need to be an additional way to manage this per site. Obviously some sites use multiple headers for other forms of auth or fingerprinting

@timothycarambat commented on GitHub (Apr 14, 2025): @agorman Absolutely, that would be the main idea here. However, I think passing auth could be tricky here since often that can be site-specific. So the issue becomes pretty wide in impact. For example, consider basic JWT `Authentication Bearer` headers. That same header is used across many sites, but the token changes per site. So there would need to be an additional way to manage this per site. Obviously some sites use multiple headers for other forms of auth or fingerprinting
yindo commented 2026-02-22 18:29:15 -05:00
Author
Owner
Copy Link
Copy Source

@agorman commented on GitHub (Apr 15, 2025):

@timothycarambat maybe I'm misunderstanding. If I pass a link and headers to /v1/document/upload-link wouldn't that be a single set of headers to a single link? So in that case the headers I send would always be site specific based on the link.

Maybe I'm taking the wrong approach. What I'm trying to do is feed it information from an internal HTTP API. I ran into a couple road blocks right away. Not being able to auth and https://github.com/Mintplex-Labs/anything-llm/issues/3625. Is there a better way to accomplish what I'm trying to do? Or maybe it's just out of scope?

@agorman commented on GitHub (Apr 15, 2025): @timothycarambat maybe I'm misunderstanding. If I pass a link and headers to `/v1/document/upload-link` wouldn't that be a single set of headers to a single link? So in that case the headers I send would always be site specific based on the link. Maybe I'm taking the wrong approach. What I'm trying to do is feed it information from an internal HTTP API. I ran into a couple road blocks right away. Not being able to auth and https://github.com/Mintplex-Labs/anything-llm/issues/3625. Is there a better way to accomplish what I'm trying to do? Or maybe it's just out of scope?
yindo commented 2026-02-22 18:29:15 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Apr 15, 2025):

If I pass a link and headers to /v1/document/upload-link wouldn't that be a single set of headers to a single link? So in that case the headers I send would always be site specific based on the link.

Correct, via the API this would be easy to handle since it would only apply per-request. I was thinking from the UI side for those who dont use the API. If we limited this functionality to just the API it is much easier to support.

As for #3625 we will probably just delete that code or add a flag to route around it. Its still useful for some peoples deployments so likely just adding a flag to work around it would be best.

@timothycarambat commented on GitHub (Apr 15, 2025): > If I pass a link and headers to /v1/document/upload-link wouldn't that be a single set of headers to a single link? So in that case the headers I send would always be site specific based on the link. Correct, via the API this would be easy to handle since it would only apply per-request. I was thinking from the UI side for those who dont use the API. If we limited this functionality to just the API it is much easier to support. As for #3625 we will probably just delete that code or add a flag to route around it. Its still useful for some peoples deployments so likely just adding a flag to work around it would be best.
yindo commented 2026-02-22 18:29:15 -05:00
Author
Owner
Copy Link
Copy Source

@agorman commented on GitHub (Apr 15, 2025):

I was just imagining a way to enter headers when fetching a website in the documents tab. In that case the headers would need to be entered per fetch and wouldn't need to be retained in the backend. I may be overlooking other places in the UI that might cause an issue though.

For #3625 a flag would be amazing.

@agorman commented on GitHub (Apr 15, 2025): I was just imagining a way to enter headers when fetching a website in the documents tab. In that case the headers would need to be entered per fetch and wouldn't need to be retained in the backend. I may be overlooking other places in the UI that might cause an issue though. For #3625 a flag would be amazing.
yindo changed title from [FEAT]: Allow custom headers for the upload-link endpoint to [GH-ISSUE #3626] [FEAT]: Allow custom headers for the upload-link endpoint 2026-06-05 14:46:00 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label core-team-only enhancement feature request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#2339
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?