Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[PR #2044] [CLOSED] 1959 filetype filters #3870

New Issue
Closed
opened 2026-02-22 18:34:40 -05:00 by yindo · 0 comments
yindo commented 2026-02-22 18:34:40 -05:00
Owner
Copy Link
Copy Source

📋 Pull Request Information

Original PR: https://github.com/Mintplex-Labs/anything-llm/pull/2044
Author: @DipFlip
Created: 8/4/2024
Status: ❌ Closed

Base: master ← Head: 1959-filetype-filters

📝 Commits (8)

  • 6d746c3 Updated the GitHubRepoLoader class to use the new import syntax and adjust the recursiveLoader method accordingly.
  • a69e63d add @langchain/community to collector package.json
  • 20e5af9 fix: Improve handling of complex ignore patterns in GitLabRepoLoader
  • 24a4d56 refactor: use ignore package for simplified ignore logic
  • 6fcf5b4 run yarn lint
  • 41499b1 Merge remote-tracking branch 'origin/master' into 1959-filetype-filters
  • b747a64 add @langchain/community@^0.2.23
  • 43606e7 Merge branch 'master' into 1959-filetype-filters

📊 Changes

4 files changed (+122 additions, -17 deletions)

View changed files

📝 collector/package.json (+2 -1)
📝 collector/utils/extensions/RepoLoader/GithubRepo/RepoLoader/index.js (+6 -7)
📝 collector/utils/extensions/RepoLoader/GitlabRepo/RepoLoader/index.js (+5 -9)
📝 collector/yarn.lock (+109 -0)

📄 Description

Pull Request Type

  • ✨ feat
  • 🐛 fix
  • ♻️ refactor
  • 💄 style
  • 🔨 chore
  • 📝 docs

Relevant Issues

resolves https://github.com/Mintplex-Labs/anything-llm/issues/1959

What is in this change?

This PR enables the Github and Gitlab data collectors to accept patterns like

*,  !*/,  !**/*.pdf

that selects only pdf files.

It fixes the issue by updating the GithubRepoCollector to use the latest l@langchain/community/document_loaders/web/github instead of the deprecated
langchain/document_loaders/web/github. Since @langchain/community 0.2.23 a bug with the GithubRepoLoader was fixed so negative gitignore patterns like !/* is now properly handled.

The GitlabRepoLoader was also not handling gitignore patterns with a ! properly. This PR makes use of the ignore package to properly check whether a file is ignored or not by the set ignorePaths.

Additional Information

Developer Validations

  • I ran yarn lint from the root of the repo & committed changes
  • Relevant documentation has been updated
  • I have tested my code functionality
  • Docker build succeeds locally

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/Mintplex-Labs/anything-llm/pull/2044 **Author:** [@DipFlip](https://github.com/DipFlip) **Created:** 8/4/2024 **Status:** ❌ Closed **Base:** `master` ← **Head:** `1959-filetype-filters` --- ### 📝 Commits (8) - [`6d746c3`](https://github.com/Mintplex-Labs/anything-llm/commit/6d746c39b27e867d5bb5f817bda7788fefcb6df0) Updated the `GitHubRepoLoader` class to use the new import syntax and adjust the `recursiveLoader` method accordingly. - [`a69e63d`](https://github.com/Mintplex-Labs/anything-llm/commit/a69e63dc1a480aaf48057fed9bff41aaa17d61bc) add @langchain/community to collector package.json - [`20e5af9`](https://github.com/Mintplex-Labs/anything-llm/commit/20e5af9f2843514e5f3c9ac1a507ca486a762158) fix: Improve handling of complex ignore patterns in GitLabRepoLoader - [`24a4d56`](https://github.com/Mintplex-Labs/anything-llm/commit/24a4d565b00793bd6bfe8076468c453063fd8535) refactor: use ignore package for simplified ignore logic - [`6fcf5b4`](https://github.com/Mintplex-Labs/anything-llm/commit/6fcf5b45d99d2c25b40d0c3f65275134af91e05d) run yarn lint - [`41499b1`](https://github.com/Mintplex-Labs/anything-llm/commit/41499b10ebb545630abcf37889ed485f9dae316f) Merge remote-tracking branch 'origin/master' into 1959-filetype-filters - [`b747a64`](https://github.com/Mintplex-Labs/anything-llm/commit/b747a6494c0d1fe18e0db620d938d38c2284e5f5) add @langchain/community@^0.2.23 - [`43606e7`](https://github.com/Mintplex-Labs/anything-llm/commit/43606e760f9f65f93d42b7425dbf7b9b6fe92e2a) Merge branch 'master' into 1959-filetype-filters ### 📊 Changes **4 files changed** (+122 additions, -17 deletions) <details> <summary>View changed files</summary> 📝 `collector/package.json` (+2 -1) 📝 `collector/utils/extensions/RepoLoader/GithubRepo/RepoLoader/index.js` (+6 -7) 📝 `collector/utils/extensions/RepoLoader/GitlabRepo/RepoLoader/index.js` (+5 -9) 📝 `collector/yarn.lock` (+109 -0) </details> ### 📄 Description ### Pull Request Type <!-- For change type, change [ ] to [x]. --> - [ ] ✨ feat - [x] 🐛 fix - [ ] ♻️ refactor - [ ] 💄 style - [ ] 🔨 chore - [ ] 📝 docs ### Relevant Issues <!-- Use "resolves #xxx" to auto resolve on merge. Otherwise, please use "connect #xxx" --> resolves https://github.com/Mintplex-Labs/anything-llm/issues/1959 ### What is in this change? <!-- Describe the changes in this PR that are impactful to the repo. --> This PR enables the Github and Gitlab data collectors to accept patterns like ```gitignore *, !*/, !**/*.pdf ``` that selects only `pdf` files. It fixes the issue by updating the GithubRepoCollector to use the latest `l@langchain/community/document_loaders/web/github` instead of the deprecated `langchain/document_loaders/web/github`. Since `@langchain/community 0.2.23` a bug with the `GithubRepoLoader` was fixed so negative gitignore patterns like `!/*` is now properly handled. The `GitlabRepoLoader` was also not handling gitignore patterns with a `!` properly. This PR makes use of the `ignore` package to properly check whether a file is ignored or not by the set `ignorePaths`. ### Additional Information <!-- Add any other context about the Pull Request here that was not captured above. --> ### Developer Validations <!-- All of the applicable items should be checked. --> - [x] I ran `yarn lint` from the root of the repo & committed changes - [x] Relevant documentation has been updated - [x] I have tested my code functionality - [ ] Docker build succeeds locally --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
yindo added the pull-request label 2026-02-22 18:34:40 -05:00
yindo closed this issue 2026-02-22 18:34:40 -05:00
yindo changed title from [PR #2044] 1959 filetype filters to [PR #2044] [CLOSED] 1959 filetype filters 2026-06-05 15:15:38 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#3870
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?