Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #1458] [FEAT]: Refreshable data sources, and question about exact method of html data vectorization. #933

New Issue
Closed
opened 2026-02-22 18:22:15 -05:00 by yindo · 4 comments
yindo commented 2026-02-22 18:22:15 -05:00
Owner
Copy Link
Copy Source

Originally created by @atljoseph on GitHub (May 19, 2024).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/1458

What would you like to see?

Be able to input a GitHub repo and GitHub pat token and have the software remember that and be able to let the user go and click refresh on it to automatically replace and reindex it. Confluence and YouTube transcripts too, along with NOT having to copy paste creds in each time. That is really cumbersome. It’s also cumbersome to have to ALWAYS re-input the files to be ignored.

Might be hard to do with files. I get that. But refresh would be a huge value add.

Lastly, I notice that webpages indexed are stored as html. Do they get indexed that way? Wouldn’t it be better to turn to pdf (headless chrome) and then read it, or read it with reader mode, and then have LLM summarize?

Originally created by @atljoseph on GitHub (May 19, 2024). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/1458 ### What would you like to see? Be able to input a GitHub repo and GitHub pat token and have the software remember that and be able to let the user go and click refresh on it to automatically replace and reindex it. Confluence and YouTube transcripts too, along with NOT having to copy paste creds in each time. That is really cumbersome. It’s also cumbersome to have to ALWAYS re-input the files to be ignored. Might be hard to do with files. I get that. But refresh would be a huge value add. Lastly, I notice that webpages indexed are stored as html. Do they get indexed that way? Wouldn’t it be better to turn to pdf (headless chrome) and then read it, or read it with reader mode, and then have LLM summarize?
yindo added the enhancementfeature request labels 2026-02-22 18:22:15 -05:00
yindo closed this issue 2026-02-22 18:22:15 -05:00
yindo commented 2026-02-22 18:22:15 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (May 19, 2024):

Lastly, I notice that webpages indexed are stored as html. Do they get indexed that way? Wouldn’t it be better to turn to pdf (headless chrome) and then read it, or read it with reader mode, and then have LLM summarize?

No, this would not be better because there is no reliable way to truly turn HTML into a PDF that does not result in a massive loss of data. When we pull in HTML we pull in the text only, not the tags plus text since a lot of that data is not useful as content anyway. Going from HTML -> PDF -> Text is duplicative because you can go from HTML -> Text with less loss.

Live file import is being handled in https://github.com/Mintplex-Labs/anything-llm/issues/1129

@timothycarambat commented on GitHub (May 19, 2024): > Lastly, I notice that webpages indexed are stored as html. Do they get indexed that way? Wouldn’t it be better to turn to pdf (headless chrome) and then read it, or read it with reader mode, and then have LLM summarize? No, this would not be better because there is no reliable way to truly turn HTML into a PDF that does not result in a massive loss of data. When we pull in HTML we pull in the text only, not the tags plus text since a lot of that data is not useful as content anyway. Going from HTML -> PDF -> Text is duplicative because you can go from HTML -> Text with less loss. Live file import is being handled in https://github.com/Mintplex-Labs/anything-llm/issues/1129
yindo commented 2026-02-22 18:22:15 -05:00
Author
Owner
Copy Link
Copy Source

@atljoseph commented on GitHub (May 19, 2024):

Ok. I’m not sure this needs closing. The primary ask is refreshability.

On Sun, May 19, 2024 at 4:34 PM Timothy Carambat @.***>
wrote:

Lastly, I notice that webpages indexed are stored as html. Do they get
indexed that way? Wouldn’t it be better to turn to pdf (headless chrome)
and then read it, or read it with reader mode, and then have LLM summarize?

No, this would not be better because there is no reliable way to truly
turn HTML into a PDF that does not result in a massive loss of data. When
we pull in HTML we pull in the text only, not the tags plus text since a
lot of that data is not useful as content anyway. Going from HTML -> PDF ->
Text is duplicative because you can go from HTML -> Text with less loss.

Live file import is being handled in #1129
https://github.com/Mintplex-Labs/anything-llm/issues/1129

—
Reply to this email directly, view it on GitHub
https://github.com/Mintplex-Labs/anything-llm/issues/1458#issuecomment-2119352054,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AF632FMMAJLHFGIW6NBE2TTZDEEF5AVCNFSM6AAAAABH6ULDKGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMJZGM2TEMBVGQ
.
You are receiving this because you authored the thread.Message ID:
@.***>

@atljoseph commented on GitHub (May 19, 2024): Ok. I’m not sure this needs closing. The primary ask is refreshability. On Sun, May 19, 2024 at 4:34 PM Timothy Carambat ***@***.***> wrote: > Lastly, I notice that webpages indexed are stored as html. Do they get > indexed that way? Wouldn’t it be better to turn to pdf (headless chrome) > and then read it, or read it with reader mode, and then have LLM summarize? > > No, this would not be better because there is no reliable way to truly > turn HTML into a PDF that does not result in a massive loss of data. When > we pull in HTML we pull in the text only, not the tags plus text since a > lot of that data is not useful as content anyway. Going from HTML -> PDF -> > Text is duplicative because you can go from HTML -> Text with less loss. > > Live file import is being handled in #1129 > <https://github.com/Mintplex-Labs/anything-llm/issues/1129> > > — > Reply to this email directly, view it on GitHub > <https://github.com/Mintplex-Labs/anything-llm/issues/1458#issuecomment-2119352054>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AF632FMMAJLHFGIW6NBE2TTZDEEF5AVCNFSM6AAAAABH6ULDKGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMJZGM2TEMBVGQ> > . > You are receiving this because you authored the thread.Message ID: > ***@***.***> >
yindo commented 2026-02-22 18:22:15 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (May 19, 2024):

Refreshing is the same concept as live-sync: https://github.com/Mintplex-Labs/anything-llm/issues/1129

Fundamentally both require the same specifications

@timothycarambat commented on GitHub (May 19, 2024): Refreshing is the same concept as live-sync: https://github.com/Mintplex-Labs/anything-llm/issues/1129 Fundamentally both require the same specifications
yindo commented 2026-02-22 18:22:16 -05:00
Author
Owner
Copy Link
Copy Source

@atljoseph commented on GitHub (May 19, 2024):

Perfect thanks

On Sun, May 19, 2024 at 6:08 PM Timothy Carambat @.***>
wrote:

Refreshing is the same concept as live-sync: #1129
https://github.com/Mintplex-Labs/anything-llm/issues/1129

Fundamentally both require the same specifications

—
Reply to this email directly, view it on GitHub
https://github.com/Mintplex-Labs/anything-llm/issues/1458#issuecomment-2119374588,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AF632FMCNU3QPIBYBON4UPDZDEPFRAVCNFSM6AAAAABH6ULDKGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMJZGM3TINJYHA
.
You are receiving this because you authored the thread.Message ID:
@.***>

@atljoseph commented on GitHub (May 19, 2024): Perfect thanks On Sun, May 19, 2024 at 6:08 PM Timothy Carambat ***@***.***> wrote: > Refreshing is the same concept as live-sync: #1129 > <https://github.com/Mintplex-Labs/anything-llm/issues/1129> > > Fundamentally both require the same specifications > > — > Reply to this email directly, view it on GitHub > <https://github.com/Mintplex-Labs/anything-llm/issues/1458#issuecomment-2119374588>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AF632FMCNU3QPIBYBON4UPDZDEPFRAVCNFSM6AAAAABH6ULDKGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMJZGM3TINJYHA> > . > You are receiving this because you authored the thread.Message ID: > ***@***.***> >
yindo changed title from [FEAT]: Refreshable data sources, and question about exact method of html data vectorization. to [GH-ISSUE #1458] [FEAT]: Refreshable data sources, and question about exact method of html data vectorization. 2026-06-05 14:38:01 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label enhancement feature request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#933
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?