Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-18 19:34:33 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[PR #5554] [CLOSED] fix(collector): strip Kufer kw-highlight rotating course blocks #5483

New Issue
Closed
opened 2026-06-05 15:21:30 -04:00 by yindo · 0 comments
yindo commented 2026-06-05 15:21:30 -04:00
Owner
Copy Link
Copy Source

📋 Pull Request Information

Original PR: https://github.com/Mintplex-Labs/anything-llm/pull/5554
Author: @Alminc91
Created: 4/29/2026
Status: ❌ Closed

Base: master ← Head: fix/collector-kw-highlight-strip

📝 Commits (10+)

  • 7f1c57a Enhance temperature control and improve OpenAI API consistency
  • c785058 newest embed state
  • 83d4c8d feat: RBAC enhancements for SaaS billing model v2.2
  • 106433b Merge upstream/master with RBAC enhancements
  • 8114bda fix: RBAC refinements for SaaS billing model
  • afb524c docs: Fix directory path in BUILD.md (anythingllm not anything-llm)
  • ea249ca docs: Add DEV_SETUP.md with local development instructions
  • 8540ea0 feat: Add multilingual TTS normalizer and STT endpoint
  • d4b7281 fix: Pin n2words to 1.21.0 for Node 18 compatibility
  • 80cb027 feat: Auto-generate weekday/month abbreviations via Intl

📊 Changes

208 files changed (+44103 additions, -681 deletions)

View changed files

➕ .claude/skills/review-code.md (+175 -0)
📝 .gitignore (+2 -0)
➕ BROWSER-TEST-DATEPICKER.md (+226 -0)
➕ BUILD.md (+67 -0)
➕ CHANGELOG.md (+272 -0)
➕ CLAUDE.md (+128 -0)
➕ DEV.md (+61 -0)
➕ DEV_SETUP.md (+119 -0)
➕ ROADMAP.md (+341 -0)
➕ ROLE_UPGRADE_PATH.md (+155 -0)
📝 collector/processLink/convert/generic.js (+108 -17)
📝 collector/processRawText/index.js (+5 -1)
📝 docker/Dockerfile (+5 -0)
📝 docker/docker-compose.yml (+18 -11)
📝 docker/docker-entrypoint.sh (+57 -3)
➕ docker/get-docker.sh (+694 -0)
➕ frontend/package-lock.json (+8282 -0)
📝 frontend/package.json (+1 -0)
📝 frontend/public/embed/anythingllm-chat-widget.min.css (+1 -1)
📝 frontend/public/embed/anythingllm-chat-widget.min.js (+5 -5)

...and 80 more files

📄 Description

Summary

Mirror of kufer-crawler PR #6. Adds [class*="kw-highlight"] to the pre-extraction strip list in getPageContent (collector/processLink/convert/generic.js).

Why

Kufer's "kw-highlight" containers rotate a random 5-9 course selection on every page load, which made every Puppeteer scrape of an upload-link page produce a fresh content hash even when nothing actually changed. This was a major source of pseudo-drift in our hourly course-refresh pipeline.

Why it's safe

The rotated courses remain reachable via:

  • Category pages
  • Course-detail pages (extractor-sourced raw-text uploads)

Verified by comparing 2 reloads (3s apart) with and without the strip — drift goes away on affected sites; unaffected sites stay unchanged.

Test plan

  • Hash-compare drift test on vhs-laufen / magdeburg / duesseldorf / intern sites confirms stabilisation
  • Container rebuild + rollout, then observe pipeline ~changed counts drop for these clients

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/Mintplex-Labs/anything-llm/pull/5554 **Author:** [@Alminc91](https://github.com/Alminc91) **Created:** 4/29/2026 **Status:** ❌ Closed **Base:** `master` ← **Head:** `fix/collector-kw-highlight-strip` --- ### 📝 Commits (10+) - [`7f1c57a`](https://github.com/Mintplex-Labs/anything-llm/commit/7f1c57a202497bc3c4aaee44d13984ce8ada2e1d) Enhance temperature control and improve OpenAI API consistency - [`c785058`](https://github.com/Mintplex-Labs/anything-llm/commit/c785058384a2e38f9bfd2b2fb15735e9979fee6f) newest embed state - [`83d4c8d`](https://github.com/Mintplex-Labs/anything-llm/commit/83d4c8d047c764c63566870f810a3b175184a536) feat: RBAC enhancements for SaaS billing model v2.2 - [`106433b`](https://github.com/Mintplex-Labs/anything-llm/commit/106433bbd2f91fa6ab8de470d8a66832322a0dc0) Merge upstream/master with RBAC enhancements - [`8114bda`](https://github.com/Mintplex-Labs/anything-llm/commit/8114bda72956d448fb1e747f1dee4e56ce722875) fix: RBAC refinements for SaaS billing model - [`afb524c`](https://github.com/Mintplex-Labs/anything-llm/commit/afb524cfb66d308a66c002eb497865082c43b134) docs: Fix directory path in BUILD.md (anythingllm not anything-llm) - [`ea249ca`](https://github.com/Mintplex-Labs/anything-llm/commit/ea249ca84372c8a3366ba866e81b254304690616) docs: Add DEV_SETUP.md with local development instructions - [`8540ea0`](https://github.com/Mintplex-Labs/anything-llm/commit/8540ea076f7102dd72c6b254b03b615e81c08a8b) feat: Add multilingual TTS normalizer and STT endpoint - [`d4b7281`](https://github.com/Mintplex-Labs/anything-llm/commit/d4b72813169ba2b2b592b20f628356c71d3a52a5) fix: Pin n2words to 1.21.0 for Node 18 compatibility - [`80cb027`](https://github.com/Mintplex-Labs/anything-llm/commit/80cb0277988639122ca54641ff04b755cb007aba) feat: Auto-generate weekday/month abbreviations via Intl ### 📊 Changes **208 files changed** (+44103 additions, -681 deletions) <details> <summary>View changed files</summary> ➕ `.claude/skills/review-code.md` (+175 -0) 📝 `.gitignore` (+2 -0) ➕ `BROWSER-TEST-DATEPICKER.md` (+226 -0) ➕ `BUILD.md` (+67 -0) ➕ `CHANGELOG.md` (+272 -0) ➕ `CLAUDE.md` (+128 -0) ➕ `DEV.md` (+61 -0) ➕ `DEV_SETUP.md` (+119 -0) ➕ `ROADMAP.md` (+341 -0) ➕ `ROLE_UPGRADE_PATH.md` (+155 -0) 📝 `collector/processLink/convert/generic.js` (+108 -17) 📝 `collector/processRawText/index.js` (+5 -1) 📝 `docker/Dockerfile` (+5 -0) 📝 `docker/docker-compose.yml` (+18 -11) 📝 `docker/docker-entrypoint.sh` (+57 -3) ➕ `docker/get-docker.sh` (+694 -0) ➕ `frontend/package-lock.json` (+8282 -0) 📝 `frontend/package.json` (+1 -0) 📝 `frontend/public/embed/anythingllm-chat-widget.min.css` (+1 -1) 📝 `frontend/public/embed/anythingllm-chat-widget.min.js` (+5 -5) _...and 80 more files_ </details> ### 📄 Description ## Summary Mirror of [kufer-crawler PR #6](https://github.com/Alminc91/kufer-crawler/pull/6). Adds `[class*="kw-highlight"]` to the pre-extraction strip list in `getPageContent` (`collector/processLink/convert/generic.js`). ## Why Kufer's "kw-highlight" containers rotate a random 5-9 course selection on every page load, which made every Puppeteer scrape of an upload-link page produce a fresh content hash even when nothing actually changed. This was a major source of pseudo-drift in our hourly course-refresh pipeline. ## Why it's safe The rotated courses remain reachable via: - Category pages - Course-detail pages (extractor-sourced raw-text uploads) Verified by comparing 2 reloads (3s apart) with and without the strip — drift goes away on affected sites; unaffected sites stay unchanged. ## Test plan - [x] Hash-compare drift test on vhs-laufen / magdeburg / duesseldorf / intern sites confirms stabilisation - [ ] Container rebuild + rollout, then observe pipeline `~changed` counts drop for these clients --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
yindo added the pull-request label 2026-06-05 15:21:30 -04:00
yindo closed this issue 2026-06-05 15:21:30 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#5483
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?