Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[PR #5188] [CLOSED] feat: opt-in query rewriting for multi-turn RAG conversations #5341

New Issue
Closed
opened 2026-06-05 15:21:02 -04:00 by yindo · 0 comments
yindo commented 2026-06-05 15:21:02 -04:00
Owner
Copy Link
Copy Source

📋 Pull Request Information

Original PR: https://github.com/Mintplex-Labs/anything-llm/pull/5188
Author: @Alminc91
Created: 3/10/2026
Status: ❌ Closed

Base: master ← Head: feat/query-rewriting-for-contextual-rag

📝 Commits (5)

  • 21a3059 feat: LLM-based query rewriting for contextual RAG in multi-turn conversations
  • 78590fd improve: selective rewriting with UNCHANGED signal and 3-layer output validation
  • a657017 improve: add universal script fallback for CJK/Thai in output validation
  • 3ed4e90 improve: simplify prompt and validation based on 250-query benchmark
  • 9f2afa1 improve: opt-in workspace setting with UI toggle, simplify prompt

📊 Changes

9 files changed (+198 additions, -6 deletions)

View changed files

➕ frontend/src/pages/WorkspaceSettings/ChatSettings/QueryRewriteMode/index.jsx (+46 -0)
📝 frontend/src/pages/WorkspaceSettings/ChatSettings/index.jsx (+5 -0)
📝 server/models/workspace.js (+10 -0)
📝 server/prisma/schema.prisma (+1 -0)
📝 server/utils/chats/apiChatHandler.js (+17 -2)
📝 server/utils/chats/embed.js (+9 -1)
📝 server/utils/chats/openaiCompatible.js (+17 -2)
📝 server/utils/chats/stream.js (+9 -1)
➕ server/utils/helpers/chat/queryRewriter.js (+84 -0)

📄 Description

Problem

In multi-turn conversations, follow-up queries like "Yes, the B1 course please" or "Tell me more about that" fail at the RAG retrieval stage. Vector search on the literal text "Yes, B1 please" returns 0 relevant results. Reranking cannot fix this — it reorders results after retrieval, and reordering 0 relevant results still yields 0 relevant results.

This is the single biggest quality gap in multi-turn RAG, and the reason every major RAG framework has adopted query rewriting as a pre-retrieval step.

Solution

A 75-line module (queryRewriter.js) that rewrites ambiguous follow-up queries into standalone search queries before vector search. Integrates via a 2-line import+call in each chat handler.

Opt-in per workspace. Disabled by default. Enable via a toggle in Chat Settings → "Query Rewriting". When disabled, zero code paths change.

How it works

  1. Gate: Only runs when chat history exists, query is ≤12 words, and the workspace has query rewriting enabled
  2. Rewrite: Sends a short prompt (~550 tokens) with the last 2 turn-pairs to the workspace LLM
  3. Use: Rewritten query goes to performSimilaritySearch. Original message is preserved for chat completion and history
  4. Fallback: On any error, the original query is used. Zero risk of degrading existing behavior

Benchmark Results

Tested on Mistral Small 24B (FP8), a small locally-hosted model, across 250 queries (6 prompt variants):

Prompt Variant Follow-ups (25) Self-contained (25) Hallucinations Overall
Final version 25/25 (100%) 21/25 (84%) 0 92%
UNCHANGED signal 17/25 (68%) 25/25 (100%) 0 84%
Reference extraction 23/25 (92%) 18/25 (72%) 0 82%
LangChain prompt ~20/25 (80%) 3/25 (12%) 5+ ~46%

The 4 self-contained "errors" in the final version are harmless paraphrases (same search results). Zero hallucinations, zero meta-text leakage across all 50 test queries.

Latency: ~260ms self-contained, ~300ms rewrites. Tested on a small on-device model.

Safety Features

Feature Behavior
Opt-in per workspace Disabled by default. Enable in Chat Settings
Env override ENABLE_QUERY_REWRITING=true sets default for all workspaces
Word count threshold Queries above 12 words skip rewriting (configurable via QUERY_REWRITE_WORD_THRESHOLD)
Verbatim detection If the LLM returns the query unchanged, original is used
Error fallback Any exception returns the original query — zero disruption
History gate First message always skips (no history to reference)

Worst case: Original query used unchanged. Cannot produce worse results than current behavior.

Changes

File Change
server/utils/helpers/chat/queryRewriter.js New — rewrite logic + prompt (~75 lines)
server/utils/chats/embed.js Import + call before vector search (+2 lines)
server/utils/chats/stream.js Import + call before vector search (+2 lines)
server/utils/chats/apiChatHandler.js Import + call before vector search (+2 lines, 2 handlers)
server/utils/chats/openaiCompatible.js Import + call before vector search (+2 lines, 2 handlers)
server/models/workspace.js Add queryRewriteMode to writable fields + validation
server/prisma/schema.prisma Add queryRewriteMode column (nullable, default "off")
frontend/.../ChatSettings/QueryRewriteMode/index.jsx New — UI toggle component (~45 lines)
frontend/.../ChatSettings/index.jsx Import + render toggle

No new dependencies. No breaking changes.

Industry Precedent

This is not experimental — it is the standard approach for multi-turn RAG:

  • LangChain — create_history_aware_retriever: LLM-based query contextualization
  • Open WebUI — Enabled by default; generates search queries from history
  • Vercel AI SDK — generateObject for query transformation before RAG
  • Amazon Bedrock Knowledge Bases — Built-in query reformulation
  • Google Vertex AI RAG — Context-aware query rewriting

Related Issues

  • #4352 (Contextual RAG)
  • #4071 (RAG retrieval misses chat history reference)
  • #4072 (follow-up)

🤖 Generated with Claude Code

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/Mintplex-Labs/anything-llm/pull/5188 **Author:** [@Alminc91](https://github.com/Alminc91) **Created:** 3/10/2026 **Status:** ❌ Closed **Base:** `master` ← **Head:** `feat/query-rewriting-for-contextual-rag` --- ### 📝 Commits (5) - [`21a3059`](https://github.com/Mintplex-Labs/anything-llm/commit/21a3059698b05702737e53a6b9cf83a70bbd67fc) feat: LLM-based query rewriting for contextual RAG in multi-turn conversations - [`78590fd`](https://github.com/Mintplex-Labs/anything-llm/commit/78590fd3e789b8ff95bda18ad5eedd5952a5289e) improve: selective rewriting with UNCHANGED signal and 3-layer output validation - [`a657017`](https://github.com/Mintplex-Labs/anything-llm/commit/a657017ed0e8cb841e4fde8d7d718f68f0284041) improve: add universal script fallback for CJK/Thai in output validation - [`3ed4e90`](https://github.com/Mintplex-Labs/anything-llm/commit/3ed4e90b52b75e7f54e63d654431da13e940c3a8) improve: simplify prompt and validation based on 250-query benchmark - [`9f2afa1`](https://github.com/Mintplex-Labs/anything-llm/commit/9f2afa16cb06f4e51df34a216486a7f1643cede4) improve: opt-in workspace setting with UI toggle, simplify prompt ### 📊 Changes **9 files changed** (+198 additions, -6 deletions) <details> <summary>View changed files</summary> ➕ `frontend/src/pages/WorkspaceSettings/ChatSettings/QueryRewriteMode/index.jsx` (+46 -0) 📝 `frontend/src/pages/WorkspaceSettings/ChatSettings/index.jsx` (+5 -0) 📝 `server/models/workspace.js` (+10 -0) 📝 `server/prisma/schema.prisma` (+1 -0) 📝 `server/utils/chats/apiChatHandler.js` (+17 -2) 📝 `server/utils/chats/embed.js` (+9 -1) 📝 `server/utils/chats/openaiCompatible.js` (+17 -2) 📝 `server/utils/chats/stream.js` (+9 -1) ➕ `server/utils/helpers/chat/queryRewriter.js` (+84 -0) </details> ### 📄 Description ## Problem In multi-turn conversations, follow-up queries like *"Yes, the B1 course please"* or *"Tell me more about that"* fail at the RAG retrieval stage. Vector search on the literal text `"Yes, B1 please"` returns **0 relevant results**. Reranking cannot fix this — it reorders results *after* retrieval, and reordering 0 relevant results still yields 0 relevant results. This is the single biggest quality gap in multi-turn RAG, and the reason every major RAG framework has adopted query rewriting as a pre-retrieval step. ## Solution A 75-line module (`queryRewriter.js`) that rewrites ambiguous follow-up queries into standalone search queries before vector search. Integrates via a 2-line import+call in each chat handler. **Opt-in per workspace.** Disabled by default. Enable via a toggle in Chat Settings → "Query Rewriting". When disabled, zero code paths change. ### How it works 1. **Gate**: Only runs when chat history exists, query is ≤12 words, and the workspace has query rewriting enabled 2. **Rewrite**: Sends a short prompt (~550 tokens) with the last 2 turn-pairs to the workspace LLM 3. **Use**: Rewritten query goes to `performSimilaritySearch`. Original message is preserved for chat completion and history 4. **Fallback**: On any error, the original query is used. Zero risk of degrading existing behavior ### Benchmark Results Tested on **Mistral Small 24B (FP8)**, a small locally-hosted model, across **250 queries** (6 prompt variants): | Prompt Variant | Follow-ups (25) | Self-contained (25) | Hallucinations | Overall | |---|---|---|---|---| | **Final version** | **25/25 (100%)** | **21/25 (84%)** | **0** | **92%** | | UNCHANGED signal | 17/25 (68%) | 25/25 (100%) | 0 | 84% | | Reference extraction | 23/25 (92%) | 18/25 (72%) | 0 | 82% | | LangChain prompt | ~20/25 (80%) | 3/25 (12%) | 5+ | ~46% | The 4 self-contained "errors" in the final version are harmless paraphrases (same search results). Zero hallucinations, zero meta-text leakage across all 50 test queries. **Latency**: ~260ms self-contained, ~300ms rewrites. Tested on a small on-device model. ### Safety Features | Feature | Behavior | |---|---| | **Opt-in per workspace** | Disabled by default. Enable in Chat Settings | | **Env override** | `ENABLE_QUERY_REWRITING=true` sets default for all workspaces | | **Word count threshold** | Queries above 12 words skip rewriting (configurable via `QUERY_REWRITE_WORD_THRESHOLD`) | | **Verbatim detection** | If the LLM returns the query unchanged, original is used | | **Error fallback** | Any exception returns the original query — zero disruption | | **History gate** | First message always skips (no history to reference) | **Worst case:** Original query used unchanged. Cannot produce worse results than current behavior. ## Changes | File | Change | |---|---| | `server/utils/helpers/chat/queryRewriter.js` | **New** — rewrite logic + prompt (~75 lines) | | `server/utils/chats/embed.js` | Import + call before vector search (+2 lines) | | `server/utils/chats/stream.js` | Import + call before vector search (+2 lines) | | `server/utils/chats/apiChatHandler.js` | Import + call before vector search (+2 lines, 2 handlers) | | `server/utils/chats/openaiCompatible.js` | Import + call before vector search (+2 lines, 2 handlers) | | `server/models/workspace.js` | Add `queryRewriteMode` to writable fields + validation | | `server/prisma/schema.prisma` | Add `queryRewriteMode` column (nullable, default "off") | | `frontend/.../ChatSettings/QueryRewriteMode/index.jsx` | **New** — UI toggle component (~45 lines) | | `frontend/.../ChatSettings/index.jsx` | Import + render toggle | No new dependencies. No breaking changes. ## Industry Precedent This is not experimental — it is the standard approach for multi-turn RAG: - **LangChain** — `create_history_aware_retriever`: LLM-based query contextualization - **Open WebUI** — Enabled by default; generates search queries from history - **Vercel AI SDK** — `generateObject` for query transformation before RAG - **Amazon Bedrock Knowledge Bases** — Built-in query reformulation - **Google Vertex AI RAG** — Context-aware query rewriting ## Related Issues - #4352 (Contextual RAG) - #4071 (RAG retrieval misses chat history reference) - #4072 (follow-up) --- 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
yindo added the pull-request label 2026-06-05 15:21:02 -04:00
yindo closed this issue 2026-06-05 15:21:02 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#5341
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?