Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-21 01:25:28 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #1798] [FEAT]: Allow users to define an include pattern for the Github Data Connector #1175

New Issue
Open
opened 2026-02-22 18:23:32 -05:00 by yindo · 0 comments
yindo commented 2026-02-22 18:23:32 -05:00
Owner
Copy Link
Copy Source

Originally created by @oleander on GitHub (Jul 1, 2024).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/1798

What would you like to see?

The current filtering feature under upload a document -> Data Connectors -> File Ignores, used for file uploads, has a few limitations and, for me, often results in unwanted documents being passed on to the LLM. The UI supports an ignore pattern, which technically can be inverted using ! to define what files to include. I mainly want to determine what files to include in the vector database rather than exclude, so I frequently use the inverse pattern.

The filtering feature is implemented by the ignore npm package and, from what I can tell, doesn't allow for nested inverts.

const ignore = require('ignore');
const fs = require('fs');
const path = require('path');

const files = [
  'README.md',
  'index.js',
  'docs/guide.md',
  'src/components/Button.md',
  'src/components/Button.js',
  'src/utils/helpers.md',
  'notes.txt',
];

const matchFiles = (patterns) => {
  const ig = ignore().add(patterns);
  return files.filter(file => !ig.ignores(file));
};

const patternsList = [
  ['*', '!.md'],
  ['*', '!*.md'],
  ['*', '!**/*.md'],
  ['*', '!.md', '!**/*.md'],
];

patternsList.forEach((patterns, index) => {
  console.log(`Test ${index + 1} with patterns: ${patterns}`);
  const matchedFiles = matchFiles(patterns);
  console.log('Matched files:', matchedFiles);
  console.log('---');
});

This results in only the root files being included.

> node main.js                                                                                                  
Test 1 with patterns: *,!.md
Matched files: []
---
Test 2 with patterns: *,!*.md
Matched files: [ 'README.md' ]
---
Test 3 with patterns: *,!**/*.md
Matched files: [ 'README.md' ]
---
Test 4 with patterns: *,!.md,!**/*.md
Matched files: [ 'README.md' ]

To simplify the UI, I suggest the following changes:

  • Allow the users to define files to be included using a glob pattern, i.e. **/**/*.md
  • Before sending the loaded data to the LLM, list all files matched by the pattern. This would allow us to test the pattern before sending it to the LLM.

What do you guys think? Would this be useful? I could look into a solution after we agree on an approach or if we decide to leave it be.

Originally created by @oleander on GitHub (Jul 1, 2024). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/1798 ### What would you like to see? The current filtering feature under `upload a document -> Data Connectors -> File Ignores`, used for file uploads, has a few limitations and, for me, often results in unwanted documents being passed on to the LLM. The UI supports an ignore pattern, which technically can be inverted using `!` to define what files to include. I mainly want to determine what files to include in the vector database rather than exclude, so I frequently use the inverse pattern. The filtering feature is implemented by the `ignore` npm package and, from what I can tell, doesn't allow for nested inverts. ```javascript const ignore = require('ignore'); const fs = require('fs'); const path = require('path'); const files = [ 'README.md', 'index.js', 'docs/guide.md', 'src/components/Button.md', 'src/components/Button.js', 'src/utils/helpers.md', 'notes.txt', ]; const matchFiles = (patterns) => { const ig = ignore().add(patterns); return files.filter(file => !ig.ignores(file)); }; const patternsList = [ ['*', '!.md'], ['*', '!*.md'], ['*', '!**/*.md'], ['*', '!.md', '!**/*.md'], ]; patternsList.forEach((patterns, index) => { console.log(`Test ${index + 1} with patterns: ${patterns}`); const matchedFiles = matchFiles(patterns); console.log('Matched files:', matchedFiles); console.log('---'); }); ``` This results in only the root files being included. ```bash > node main.js Test 1 with patterns: *,!.md Matched files: [] --- Test 2 with patterns: *,!*.md Matched files: [ 'README.md' ] --- Test 3 with patterns: *,!**/*.md Matched files: [ 'README.md' ] --- Test 4 with patterns: *,!.md,!**/*.md Matched files: [ 'README.md' ] ``` To simplify the UI, I suggest the following changes: * Allow the users to define files to be included using a glob pattern, i.e. `**/**/*.md` * Before sending the loaded data to the LLM, list all files matched by the pattern. This would allow us to test the pattern before sending it to the LLM. What do you guys think? Would this be useful? I could look into a solution after we agree on an approach or if we decide to leave it be.
yindo added the enhancementfeature request labels 2026-02-22 18:23:32 -05:00
yindo changed title from [FEAT]: Allow users to define an include pattern for the Github Data Connector to [GH-ISSUE #1798] [FEAT]: Allow users to define an include pattern for the Github Data Connector 2026-06-05 14:39:22 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label enhancement feature request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#1175
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?