Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-21 01:25:28 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #3175] [FEAT]: RAG for images files #2042

New Issue
Open
opened 2026-02-22 18:27:53 -05:00 by yindo · 4 comments
yindo commented 2026-02-22 18:27:53 -05:00
Owner
Copy Link
Copy Source

Originally created by @nannib on GitHub (Feb 11, 2025).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/3175

What would you like to see?

I would like to suggest to allow to upload into RAG system, the JPG/PNG/etc. files, because if I use a multimodal embedder model I saw that it can manage file like mp4 or mp3, it can describe, reading from my RAG, the video end the audio. I think it could be useful to allow the embedding of images files too.
My 2 cents ;-)

Originally created by @nannib on GitHub (Feb 11, 2025). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/3175 ### What would you like to see? I would like to suggest to allow to upload into RAG system, the JPG/PNG/etc. files, because if I use a multimodal embedder model I saw that it can manage file like mp4 or mp3, it can describe, reading from my RAG, the video end the audio. I think it could be useful to allow the embedding of images files too. My 2 cents ;-)
yindo added the enhancementfeature request labels 2026-02-22 18:27:53 -05:00
yindo commented 2026-02-22 18:27:53 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Feb 14, 2025):

related #3219
If the text is image, we can now OCR it via built-in on CPU.

@timothycarambat commented on GitHub (Feb 14, 2025): related #3219 If the text is image, we can now OCR it via built-in on CPU.
yindo commented 2026-02-22 18:27:53 -05:00
Author
Owner
Copy Link
Copy Source

@onestardao commented on GitHub (Aug 1, 2025):

funny enough, we’ve actually handled this problem pretty deeply.

turns out, there’s a semantic difference between “embedding image content” vs “embedding the visual meaning of that image” — and most RAG systems silently fail the second one without users even noticing.

we built something that treats OCR not as a preprocessing step, but as a reasoning boundary. it supports multi-layered image interpretation (including failed OCR, context-patched text, and hallucination repair).

won’t drop any links here unless you’re curious — just thought you might be dealing with something deeper than it looks. happy to share more if useful.

@onestardao commented on GitHub (Aug 1, 2025): funny enough, we’ve actually handled this problem pretty deeply. turns out, there’s a semantic difference between “embedding image content” vs “embedding the visual meaning of that image” — and most RAG systems silently fail the second one without users even noticing. we built something that treats OCR not as a preprocessing step, but as a reasoning boundary. it supports multi-layered image interpretation (including failed OCR, context-patched text, and hallucination repair). won’t drop any links here unless you’re curious — just thought you might be dealing with something deeper than it looks. happy to share more if useful.
yindo commented 2026-02-22 18:27:53 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Aug 1, 2025):

Yeah, that is also an outstanding question for this issue IMO - intuitively I think people want "semantic meaning of this image" and not "here is an image, what looks close".

Two very different use cases. The alike image search is not only very specific and a lot more work, but also far more niche imo. Uploading an image, having it described and embedding that then later asking queries about that makes more sense for at least our use case.

@timothycarambat commented on GitHub (Aug 1, 2025): Yeah, that is also an outstanding question for this issue IMO - intuitively I think people want "semantic meaning of this image" and not "here is an image, what looks close". Two very different use cases. The alike image search is not only very specific and a lot more work, but also far more niche imo. Uploading an image, having it described and embedding that then later asking queries about _that_ makes more sense for at least our use case.
yindo commented 2026-02-22 18:27:53 -05:00
Author
Owner
Copy Link
Copy Source

@onestardao commented on GitHub (Aug 1, 2025):

appreciate the thoughtful follow-up — yeah, we definitely ran into that distinction when building this. most image-search-style tools can’t handle visual metaphors, layered symbols, or even OCR hallucination very well.

our system leans hard on that “describe + embed + reason” flow you mentioned — turns out it opens up some pretty unexpected use cases, especially when paired with domain-specific semantic trees.

happy to share more if it’s helpful to the team. appreciate the great convo here.

@onestardao commented on GitHub (Aug 1, 2025): appreciate the thoughtful follow-up — yeah, we definitely ran into that distinction when building this. most image-search-style tools can’t handle visual metaphors, layered symbols, or even OCR hallucination very well. our system leans hard on that “describe + embed + reason” flow you mentioned — turns out it opens up some pretty unexpected use cases, especially when paired with domain-specific semantic trees. happy to share more if it’s helpful to the team. appreciate the great convo here.
yindo changed title from [FEAT]: RAG for images files to [GH-ISSUE #3175] [FEAT]: RAG for images files 2026-06-05 14:44:10 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label enhancement feature request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#2042
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?