Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #3538] [BUG]: Response not retrieved exactly as in the uploaded JSON file #2280

New Issue
Closed
opened 2026-02-22 18:28:58 -05:00 by yindo · 1 comment
yindo commented 2026-02-22 18:28:58 -05:00
Owner
Copy Link
Copy Source

Originally created by @johnec2 on GitHub (Mar 25, 2025).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/3538

How are you running AnythingLLM?

Local development

What happened?

Hello,

I am currently using the following stack with AnythingLLM:

AI Model: Llama 3.2: 1b

AI Model Provider: Ollama (version 0.5.12)

Vector DB: Qdrant (version 1.13.4)

Embed Model: nomic-embed-text:latest

UI Tool: AnythingLLM (version 0.2.0)

Configuration:

Chunk Size: 2000

Chunk Overlap: 200

Threshold Similarity Score: >= 0.25

I have a JSON file that contains structured questions and answers. After uploading the file to AnythingLLM, the responses are not being retrieved exactly as they appear in the file. Instead, additional information is being added, or generalized answers are being displayed.

The JSON file is structured as follows:

{
"id": "01",
"module": "Facility",
"category": "Add Facility to Complex",
"role": ["admin", "team staff"],
"questions": [
"How can I add facilities to a complex?"
],
"answer": "Facilities can be added to a complex either by creating a new complex or for an existing complex. To do so: Navigate to Facilities > All Complexes from the left menu. From the list, choose the complex to add facility. In the Actions column, click the vertical dots and select Update Complex. In the complex details page, select the Facilities tab. In the Facilities tab, select either Add 1 Facility or Add 5 Facilities to add facilities to the complex. While adding a facility, choose either Create New Facility or Select an Existing Facility. Click the Save button. Note: When creating a new facility inside a complex, the facility will inherit the complex properties (e.g., details, address, and timings). These properties can later be updated individually through the Update Facility page.",
"type": "verbatim"
}
Despite specifying the "type": "verbatim", the response is still being paraphrased. Additionally, the introductory portion of the answer, "Facilities can be added to a complex either by creating a new complex or for an existing complex...", is missing. To address this, I added an introduction field to the data like so:

{
"id": "01",
"module": "Facility",
"category": "Add Facility to Complex",
"role": ["admin", "team staff"],
"questions": [
"How can I add facilities to a complex?"
],
"answer": "To do so: Navigate to Facilities > All Complexes from the left menu. From the list, choose the complex to add facility. In the Actions column, click the vertical dots and select Update Complex. In the complex details page, select the Facilities tab. In the Facilities tab, select either Add 1 Facility or Add 5 Facilities to add facilities to the complex. While adding a facility, choose either Create New Facility or Select an Existing Facility. Click the Save button. Note: When creating a new facility inside a complex, the facility will inherit the complex properties (e.g., details, address, and timings). These properties can later be updated individually through the Update Facility page.",
"introduction": "Facilities can be added to a complex either by creating a new complex or for an existing complex.",
"type": "verbatim"
}
However, this did not change the result. I also modified the prompt in Chat Settings > Prompt > Chat Mode > Query to the following:

"You are a system that provides answers from a document. For entries with "type": "verbatim", output the "introduction" field (if present) followed by the "answer" field, exactly as written. Do not paraphrase, summarize, or reformat the text (e.g., do not convert paragraphs to lists). Do not add any additional text beyond what is in the "introduction" and "answer" fields, such as extra introductions or conclusions. Preserve all whitespace, punctuation, and structure as it appears in the file."

Unfortunately, this also did not resolve the issue.

Could you provide a solution to ensure that the responses are fetched exactly as they appear in the file, without any modifications or added information?

Are there known steps to reproduce?

No response

Originally created by @johnec2 on GitHub (Mar 25, 2025). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/3538 ### How are you running AnythingLLM? Local development ### What happened? Hello, I am currently using the following stack with AnythingLLM: AI Model: Llama 3.2: 1b AI Model Provider: Ollama (version 0.5.12) Vector DB: Qdrant (version 1.13.4) Embed Model: nomic-embed-text:latest UI Tool: AnythingLLM (version 0.2.0) Configuration: Chunk Size: 2000 Chunk Overlap: 200 Threshold Similarity Score: >= 0.25 I have a JSON file that contains structured questions and answers. After uploading the file to AnythingLLM, the responses are not being retrieved exactly as they appear in the file. Instead, additional information is being added, or generalized answers are being displayed. The JSON file is structured as follows: { "id": "01", "module": "Facility", "category": "Add Facility to Complex", "role": ["admin", "team staff"], "questions": [ "How can I add facilities to a complex?" ], "answer": "Facilities can be added to a complex either by creating a new complex or for an existing complex. To do so: Navigate to Facilities > All Complexes from the left menu. From the list, choose the complex to add facility. In the Actions column, click the vertical dots and select Update Complex. In the complex details page, select the Facilities tab. In the Facilities tab, select either Add 1 Facility or Add 5 Facilities to add facilities to the complex. While adding a facility, choose either Create New Facility or Select an Existing Facility. Click the Save button. Note: When creating a new facility inside a complex, the facility will inherit the complex properties (e.g., details, address, and timings). These properties can later be updated individually through the Update Facility page.", "type": "verbatim" } Despite specifying the "type": "verbatim", the response is still being paraphrased. Additionally, the introductory portion of the answer, "Facilities can be added to a complex either by creating a new complex or for an existing complex...", is missing. To address this, I added an introduction field to the data like so: { "id": "01", "module": "Facility", "category": "Add Facility to Complex", "role": ["admin", "team staff"], "questions": [ "How can I add facilities to a complex?" ], "answer": "To do so: Navigate to Facilities > All Complexes from the left menu. From the list, choose the complex to add facility. In the Actions column, click the vertical dots and select Update Complex. In the complex details page, select the Facilities tab. In the Facilities tab, select either Add 1 Facility or Add 5 Facilities to add facilities to the complex. While adding a facility, choose either Create New Facility or Select an Existing Facility. Click the Save button. Note: When creating a new facility inside a complex, the facility will inherit the complex properties (e.g., details, address, and timings). These properties can later be updated individually through the Update Facility page.", "introduction": "Facilities can be added to a complex either by creating a new complex or for an existing complex.", "type": "verbatim" } However, this did not change the result. I also modified the prompt in **Chat Settings > Prompt > Chat Mode > Query** to the following: "You are a system that provides answers from a document. For entries with "type": "verbatim", output the "introduction" field (if present) followed by the "answer" field, exactly as written. Do not paraphrase, summarize, or reformat the text (e.g., do not convert paragraphs to lists). Do not add any additional text beyond what is in the "introduction" and "answer" fields, such as extra introductions or conclusions. Preserve all whitespace, punctuation, and structure as it appears in the file." Unfortunately, this also did not resolve the issue. Could you provide a solution to ensure that the responses are fetched exactly as they appear in the file, without any modifications or added information? ### Are there known steps to reproduce? _No response_
yindo added the possible bug label 2026-02-22 18:28:58 -05:00
yindo closed this issue 2026-02-22 18:28:58 -05:00
yindo commented 2026-02-22 18:28:58 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Mar 26, 2025):

Your model is doing the paraphrasing, not AnythingLLM.

Llama 3.2: 1b

This model is very small and unless you specified pulling Q8 in ollama ollama pull llama3.2:1b-text-q8_0 you are using the q4 version of the model. So now not only do you have a small param model, but is is also compressed heavily. So it is no wonder why it does not follow instructions!

Use a larger param model at Q4, or attempt to pull the Q8 version of 3.2:1b - I do think that even the 1B @ Q8 will still perform poorly here though. I would certainly recommend a larger model for this, smaller models almost always ignore specific prompt instructions.

If we have the context needed in the window, it is solely up to the model to then return the right information. That being said, verbatim recall is not an LLM strong suite. It sounds like you are doing just retrieval for RAG and dont even need an LLM in the middle though?

@timothycarambat commented on GitHub (Mar 26, 2025): Your model is doing the paraphrasing, not AnythingLLM. > Llama 3.2: 1b This model is _very small_ and unless you specified pulling `Q8` in ollama `ollama pull llama3.2:1b-text-q8_0` you are using the `q4` version of the model. So now not only do you have a small param model, but is is also compressed heavily. So it is no wonder why it does not follow instructions! Use a larger param model at Q4, or attempt to pull the Q8 version of 3.2:1b - I do think that even the 1B @ Q8 will still perform poorly here though. I would certainly recommend a larger model for this, smaller models almost always ignore specific prompt instructions. If we have the context needed in the window, it is solely up to the model to then return the right information. That being said, verbatim recall is not an LLM strong suite. It sounds like you are doing just retrieval for RAG and dont even need an LLM in the middle though?
yindo changed title from [BUG]: Response not retrieved exactly as in the uploaded JSON file to [GH-ISSUE #3538] [BUG]: Response not retrieved exactly as in the uploaded JSON file 2026-06-05 14:45:37 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label possible bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#2280
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?