Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 22:23:50 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #1551] RAG implementation with user-roles getting different results? #1001

New Issue
Closed
opened 2026-02-22 18:22:36 -05:00 by yindo · 5 comments
yindo commented 2026-02-22 18:22:36 -05:00
Owner
Copy Link
Copy Source

Originally created by @thebaldgeek on GitHub (May 27, 2024).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/1551

How are you running AnythingLLM?

Docker (local)

What happened?

Trained system from about 100 pdf and text documents. Using default and built-in database. Windows 11. Docker image pulled around 18 hours ago.

Admin user gets answers from trained docs as expected.
Non-admin users get raw Ollama llama3.0 LLM answers with correct citations.
Tested with exact same questions and other basic questions. 100% of the time the Admin user gets the correct answer text that shows the trained docs have been accessed and 4 citations contain parts of the answer text as expected.
Default user gets raw LLM answer (often comically wrong) with 'correct' 4 citations that contain question text listed (for those questions that are the same).

Tested in Firefox, Chrome and Edge, both regular browser and incognito sessions.
User is set as 'Default'. User is added to workspace.

Work around is to have all users log in as Admins and requested not to change any settings.

Are there known steps to reproduce?

No response

Originally created by @thebaldgeek on GitHub (May 27, 2024). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/1551 ### How are you running AnythingLLM? Docker (local) ### What happened? Trained system from about 100 pdf and text documents. Using default and built-in database. Windows 11. Docker image pulled around 18 hours ago. Admin user gets answers from trained docs as expected. Non-admin users get raw Ollama llama3.0 LLM answers with correct citations. Tested with exact same questions and other basic questions. 100% of the time the Admin user gets the correct answer text that shows the trained docs have been accessed and 4 citations contain parts of the answer text as expected. Default user gets raw LLM answer (often comically wrong) with 'correct' 4 citations that contain question text listed (for those questions that are the same). Tested in Firefox, Chrome and Edge, both regular browser and incognito sessions. User is set as 'Default'. User is added to workspace. Work around is to have all users log in as Admins and requested not to change any settings. ### Are there known steps to reproduce? _No response_
yindo added the core-team-onlyinvestigating labels 2026-02-22 18:22:36 -05:00
yindo closed this issue 2026-02-22 18:22:36 -05:00
yindo commented 2026-02-22 18:22:36 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (May 27, 2024):

This is not how the implementation of the RAG system works, or should work. Everyone has the same access to documents in the same workspace, so sending a prompt in a workspace the only user-specific information is the current chat/thread history sent by that specific user.

I think there is either a misunderstanding or some other detail missing from this issue that is leading to worse RAG performance at this time.

This part:

Default user gets raw LLM answer (often comically wrong) with 'correct' 4 citations that contain question text listed (for those questions that are the same).

is quite odd as the citation should not exist at all unless it was explicitly used in the response generation 🤔

@timothycarambat commented on GitHub (May 27, 2024): This is not how the implementation of the RAG system works, or should work. Everyone has the same access to documents in the same workspace, so sending a prompt in a workspace the only user-specific information is the current chat/thread history sent by that specific user. I think there is either a misunderstanding or some other detail missing from this issue that is leading to worse RAG performance at this time. This part: > Default user gets raw LLM answer (often comically wrong) with 'correct' 4 citations that contain question text listed (for those questions that are the same). is quite odd as the citation should not exist at all unless it was explicitly used in the response generation 🤔
yindo commented 2026-02-22 18:22:37 -05:00
Author
Owner
Copy Link
Copy Source

@omaralagouz commented on GitHub (Jul 29, 2024):

I have a similar issue where I've added documents to a workspace that have all been embedded and all seems to be fine, but when a question is asked, RAG is never used to answer the questions. Is there a specific configuration for this that i've missed? I tried with many users and with a different thread in the same workspace but still no luck in getting RAG to work at all. (I'm using groq with Llama 8b as my LLM)

@omaralagouz commented on GitHub (Jul 29, 2024): I have a similar issue where I've added documents to a workspace that have all been embedded and all seems to be fine, but when a question is asked, RAG is never used to answer the questions. Is there a specific configuration for this that i've missed? I tried with many users and with a different thread in the same workspace but still no luck in getting RAG to work at all. (I'm using groq with Llama 8b as my LLM)
yindo commented 2026-02-22 18:22:37 -05:00
Author
Owner
Copy Link
Copy Source

@omaralagouz commented on GitHub (Jul 29, 2024):

Okay, problem solved, it happened to be due to the 'Document similarity threshold' setting i had which was [> .75]. All citations that are returned are almost always less than .75, so changing it to [>.50] fixed my issue completely.

@omaralagouz commented on GitHub (Jul 29, 2024): Okay, problem solved, it happened to be due to the 'Document similarity threshold' setting i had which was [> .75]. All citations that are returned are almost always less than .75, so changing it to [>.50] fixed my issue completely.
yindo commented 2026-02-22 18:22:37 -05:00
Author
Owner
Copy Link
Copy Source

@thebaldgeek commented on GitHub (Jul 29, 2024):

I am stunned that this bug is closed as 'not planned'.
Since I discovered this issue on May 27, my team of 20 people have been in limbo.
Since they cant log in as anything other than admin, they have been emailing me their LLM questions, I copy/paste into the interface, and copy/paste/email the results back while we wait for this function to be fixed.
Since its now clear that multi user accounts is not a planed feature, I feel the rug has been pulled out from under myself and the entire team.
I hope the docs and features etc for this project are updated and made clear that everyone must log in as ADMIN.

@thebaldgeek commented on GitHub (Jul 29, 2024): I am stunned that this bug is closed as 'not planned'. Since I discovered this issue on May 27, my team of 20 people have been in limbo. Since they cant log in as anything other than admin, they have been emailing me their LLM questions, I copy/paste into the interface, and copy/paste/email the results back while we wait for this function to be fixed. Since its now clear that multi user accounts is not a planed feature, I feel the rug has been pulled out from under myself and the entire team. I hope the docs and features etc for this project are updated and made clear that everyone must log in as ADMIN.
yindo commented 2026-02-22 18:22:37 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jul 29, 2024):

Since its now clear that multi user accounts is not a planed feature, I feel the rug has been pulled out from under myself and the entire team.

We have multi-user accounts.

The issue was closed because the user said it was solved by changing their similarity score.

@thebaldgeek If you have made everyone an admin you are doing something very wrong. The queries someone sends have zero impact on what documents they can see. If you are able to provide any reproducible example of the same query being sent in the same workspace and the only difference being a user role - I would like to review it myself.

Documents are not even partitioned by userspace or role, everyone chats with the same documents in a workspace regardless of role. The role simply limits who can manage the documents in a workspace. If you want users to be able to make their own workspaces, make them a manager, not an admin. Then they can make their own workspaces and upload their own documents.

@timothycarambat commented on GitHub (Jul 29, 2024): > Since its now clear that multi user accounts is not a planed feature, I feel the rug has been pulled out from under myself and the entire team. We have multi-user accounts. The issue was closed because the user said it was solved by changing their similarity score. @thebaldgeek If you have made everyone an admin you are doing something very wrong. The queries someone sends have _zero impact on what documents_ they can see. If you are able to provide any reproducible example of the same query being sent in the same workspace and the only difference being a user role - I would like to review it myself. Documents are not even partitioned by userspace or role, everyone chats with the same documents in a workspace regardless of role. The role simply limits who can _manage_ the documents in a workspace. If you want users to be able to make their own workspaces, make them a manager, not an admin. Then they can make their own workspaces and upload their own documents.
yindo changed title from RAG implementation with user-roles getting different results? to [GH-ISSUE #1551] RAG implementation with user-roles getting different results? 2026-06-05 14:38:25 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label core-team-only investigating
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#1001
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?