Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[PR #5747] [MERGED] feat: make document sync stale-after interval configurable #5554

New Issue
Closed
opened 2026-06-05 15:21:43 -04:00 by yindo · 0 comments
yindo commented 2026-06-05 15:21:43 -04:00
Owner
Copy Link
Copy Source

📋 Pull Request Information

Original PR: https://github.com/Mintplex-Labs/anything-llm/pull/5747
Author: @sanidhyasin
Created: 6/3/2026
Status: ✅ Merged
Merged: 6/3/2026
Merged by: @timothycarambat

Base: master ← Head: feat/configurable-document-sync-frequency

📝 Commits (2)

  • ce059f4 feat: make document sync stale-after interval configurable
  • 73d7291 Merge branch 'master' into feat/configurable-document-sync-frequency

📊 Changes

4 files changed (+69 additions, -1 deletions)

View changed files

📝 docker/.env.example (+5 -0)
📝 server/.env.example (+5 -0)
➕ server/__tests__/models/documentSyncQueue.test.js (+41 -0)
📝 server/models/documentSyncQueue.js (+18 -1)

📄 Description

Pull Request Type

  • ✨ feat (New feature)
  • 🐛 fix (Bug fix)
  • ♻️ refactor (Code refactoring without changing behavior)
  • 💄 style (UI style changes)
  • 🔨 chore (Build, CI, maintenance)
  • 📝 docs (Documentation updates)

Relevant Issues

resolves #3988

Description

The document sync/watch background worker hardcoded its stale-after interval to 7 days (604800000ms) in server/models/documentSyncQueue.js. As noted in #3988, this is too infrequent for operational documents that change often, and self-hosters had no way to tune it short of patching the source.

This PR makes the interval configurable via a new optional DOCUMENT_SYNC_STALE_AFTER_MS environment variable, keeping the existing 7-day behavior as the default.

Changes

  • defaultStaleAfter is now a getter that reads DOCUMENT_SYNC_STALE_AFTER_MS:
    • falls back to the 7-day default when the value is unset, non-numeric, or non-positive;
    • enforces a 1-hour minimum to guard against overloading embedders by re-syncing too frequently (the concern raised by @timothycarambat in the issue thread).
  • watch() now persists staleAfterMs on the queue record at creation time, so the configured interval governs recurring syncs (via calcNextSync), not just the first one. Previously recurring syncs always fell back to the Prisma column default of 7 days.
  • Documented the variable in server/.env.example and docker/.env.example.

Additional Information

This follows the existing env-var pattern used for COMETAPI_LLM_TIMEOUT_MS (parse → validate → clamp → default).

Developer Validations

  • I ran yarn lint from the root of the repo & committed changes
  • Relevant documentation has been updated (if applicable)
  • I have tested my code functionality
  • Docker build succeeds locally

Testing notes: Added server/__tests__/models/documentSyncQueue.test.js covering the default, valid override, sub-minimum clamping, and invalid/non-positive fallbacks. The clamping/fallback logic was verified locally; a full monorepo install for the complete jest/lint toolchain was skipped as heavy.

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/Mintplex-Labs/anything-llm/pull/5747 **Author:** [@sanidhyasin](https://github.com/sanidhyasin) **Created:** 6/3/2026 **Status:** ✅ Merged **Merged:** 6/3/2026 **Merged by:** [@timothycarambat](https://github.com/timothycarambat) **Base:** `master` ← **Head:** `feat/configurable-document-sync-frequency` --- ### 📝 Commits (2) - [`ce059f4`](https://github.com/Mintplex-Labs/anything-llm/commit/ce059f4cb8912a517c33aefdb508087238add882) feat: make document sync stale-after interval configurable - [`73d7291`](https://github.com/Mintplex-Labs/anything-llm/commit/73d72915945db5315c59998f19053fe59399fe95) Merge branch 'master' into feat/configurable-document-sync-frequency ### 📊 Changes **4 files changed** (+69 additions, -1 deletions) <details> <summary>View changed files</summary> 📝 `docker/.env.example` (+5 -0) 📝 `server/.env.example` (+5 -0) ➕ `server/__tests__/models/documentSyncQueue.test.js` (+41 -0) 📝 `server/models/documentSyncQueue.js` (+18 -1) </details> ### 📄 Description ### Pull Request Type - [x] ✨ feat (New feature) - [ ] 🐛 fix (Bug fix) - [ ] ♻️ refactor (Code refactoring without changing behavior) - [ ] 💄 style (UI style changes) - [ ] 🔨 chore (Build, CI, maintenance) - [ ] 📝 docs (Documentation updates) ### Relevant Issues resolves #3988 ### Description The document sync/watch background worker hardcoded its stale-after interval to 7 days (`604800000ms`) in `server/models/documentSyncQueue.js`. As noted in #3988, this is too infrequent for operational documents that change often, and self-hosters had no way to tune it short of patching the source. This PR makes the interval configurable via a new optional `DOCUMENT_SYNC_STALE_AFTER_MS` environment variable, keeping the existing 7-day behavior as the default. **Changes** - `defaultStaleAfter` is now a getter that reads `DOCUMENT_SYNC_STALE_AFTER_MS`: - falls back to the 7-day default when the value is unset, non-numeric, or non-positive; - enforces a **1-hour minimum** to guard against overloading embedders by re-syncing too frequently (the concern raised by @timothycarambat in the issue thread). - `watch()` now persists `staleAfterMs` on the queue record at creation time, so the configured interval governs **recurring** syncs (via `calcNextSync`), not just the first one. Previously recurring syncs always fell back to the Prisma column default of 7 days. - Documented the variable in `server/.env.example` and `docker/.env.example`. ### Additional Information This follows the existing env-var pattern used for `COMETAPI_LLM_TIMEOUT_MS` (parse → validate → clamp → default). ### Developer Validations - [ ] I ran `yarn lint` from the root of the repo & committed changes - [x] Relevant documentation has been updated (if applicable) - [x] I have tested my code functionality - [ ] Docker build succeeds locally > Testing notes: Added `server/__tests__/models/documentSyncQueue.test.js` covering the default, valid override, sub-minimum clamping, and invalid/non-positive fallbacks. The clamping/fallback logic was verified locally; a full monorepo install for the complete jest/lint toolchain was skipped as heavy. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
yindo added the pull-request label 2026-06-05 15:21:43 -04:00
yindo closed this issue 2026-06-05 15:21:43 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#5554
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?