Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 22:23:50 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[PR #4692] [CLOSED] # Document Enrichment Pipeline Enhancement #4718

New Issue
Closed
opened 2026-02-22 18:36:23 -05:00 by yindo · 0 comments
yindo commented 2026-02-22 18:36:23 -05:00
Owner
Copy Link
Copy Source

📋 Pull Request Information

Original PR: https://github.com/Mintplex-Labs/anything-llm/pull/4692
Author: @Escanor4323
Created: 11/26/2025
Status: ❌ Closed

Base: master ← Head: ECP-HRAG

📝 Commits (10+)

📊 Changes

64 files changed (+8491 additions, -1323 deletions)

View changed files

➕ .github/workflows/deploy-infra.yml (+57 -0)
📝 .github/workflows/scratch/build-and-push-image-semver.yaml (+0 -0)
📝 .github/workflows/scratch/build-and-push-image.yaml (+0 -0)
📝 .github/workflows/scratch/check-translations.yaml (+0 -0)
📝 .github/workflows/scratch/dev-build.yaml (+0 -0)
📝 .gitignore (+2 -0)
📝 README.md (+11 -0)
📝 cloud-deployments/aws/cloudformation/cloudformation_create_anythingllm.json (+297 -184)
📝 cloud-deployments/gcp/deployment/gcp_deploy_anything_llm.yaml (+3 -3)
➕ cloud-deployments/helm/charts/anythingllm/.helmignore (+23 -0)
➕ cloud-deployments/helm/charts/anythingllm/Chart.yaml (+7 -0)
➕ cloud-deployments/helm/charts/anythingllm/README.md (+149 -0)
➕ cloud-deployments/helm/charts/anythingllm/README.md.gotmpl (+103 -0)
➕ cloud-deployments/helm/charts/anythingllm/templates/NOTES.txt (+28 -0)
➕ cloud-deployments/helm/charts/anythingllm/templates/_helpers.tpl (+62 -0)
➕ cloud-deployments/helm/charts/anythingllm/templates/configmap.yaml (+10 -0)
➕ cloud-deployments/helm/charts/anythingllm/templates/deployment.yaml (+83 -0)
➕ cloud-deployments/helm/charts/anythingllm/templates/extra-objects.yaml (+4 -0)
➕ cloud-deployments/helm/charts/anythingllm/templates/ingress.yaml (+61 -0)
➕ cloud-deployments/helm/charts/anythingllm/templates/pvc.yaml (+33 -0)

...and 44 more files

📄 Description

Overview

This PR introduces an enhanced document processing pipeline that improves embedding quality through contextual enrichment and external OCR integration capabilities.

Key Features

Document Enrichment

  • Implements a one-time enrichment process that adds contextual metadata to document chunks during embedding
  • Enriches chunks with position information and document context to improve retrieval accuracy
  • Configurable enrichment settings via system configuration

External OCR Integration

  • Adds API support for ingesting OCR data from external sources
  • Supports structured OCR field data with flexible field mapping
  • Maintains backward compatibility with existing document processing workflows

Enhanced Error Handling

  • Improves error handling and validation throughout the vectorization pipeline
  • Adds comprehensive error logging for better debugging
  • Fixes cache logic bug in Zilliz provider

TextSplitter Enhancements

  • Extends metadata support to include additional document fields
  • Maintains existing functionality while adding new capabilities

Technical Details

New Modules

  • server/utils/documentEnrichment.js - Core enrichment logic
  • server/utils/ocrFieldParser.js - OCR field parsing utilities

Modified Components

  • Document upload endpoints now accept external OCR data
  • Zilliz vector provider enhanced with enrichment pipeline integration
  • TextSplitter extended with additional metadata fields
  • Improved error handling across document processing flow

API Changes

  • POST /v1/document/upload and POST /v1/document/upload/:folderName now accept optional externalOCRFields parameter

Testing

  • Verified with existing document processing workflows
  • Tested error handling and edge cases
  • Validated backward compatibility

Notes

  • This enhancement is designed to work seamlessly with existing document processing
  • No breaking changes to existing APIs
  • Enrichment can be configured or disabled via system settings

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/Mintplex-Labs/anything-llm/pull/4692 **Author:** [@Escanor4323](https://github.com/Escanor4323) **Created:** 11/26/2025 **Status:** ❌ Closed **Base:** `master` ← **Head:** `ECP-HRAG` --- ### 📝 Commits (10+) - [`0ee8676`](https://github.com/Mintplex-Labs/anything-llm/commit/0ee867678955321644ba10df0e494cff95839e8c) deployment changes - [`e1e600f`](https://github.com/Mintplex-Labs/anything-llm/commit/e1e600f7f0f07fa9ed4f4b6b4812ca1ac3d088a2) moved build actions - [`755337d`](https://github.com/Mintplex-Labs/anything-llm/commit/755337d2a2d88364adb4237816e4d6122a45116b) changed main to master - [`30fc05f`](https://github.com/Mintplex-Labs/anything-llm/commit/30fc05f5333bcc68f028f0dd9414f8da4f3010cc) changes to deployment pipeline - [`6b81096`](https://github.com/Mintplex-Labs/anything-llm/commit/6b810967c1fbb1560c343a05d70e4a8acc2bca41) removed yml - [`43fdac9`](https://github.com/Mintplex-Labs/anything-llm/commit/43fdac9ae238fdc76cd8b0d8971a7b941fe61c33) Update deploy-infra.yml - [`ad0f017`](https://github.com/Mintplex-Labs/anything-llm/commit/ad0f017942d3da3e6d545f35572d896404740ff8) Update deploy-infra.yml - [`005f292`](https://github.com/Mintplex-Labs/anything-llm/commit/005f292cc862f80c1671784bcc2e6ea337672321) update deletion and update policy - [`243332a`](https://github.com/Mintplex-Labs/anything-llm/commit/243332a984a9f04ae1953a4bd1c6597047c42427) added key pem for ssh into instance - [`1fd884e`](https://github.com/Mintplex-Labs/anything-llm/commit/1fd884e6dd9874303fc2fa5d343e251a7ef64a6f) added read access to ssm in vm ### 📊 Changes **64 files changed** (+8491 additions, -1323 deletions) <details> <summary>View changed files</summary> ➕ `.github/workflows/deploy-infra.yml` (+57 -0) 📝 `.github/workflows/scratch/build-and-push-image-semver.yaml` (+0 -0) 📝 `.github/workflows/scratch/build-and-push-image.yaml` (+0 -0) 📝 `.github/workflows/scratch/check-translations.yaml` (+0 -0) 📝 `.github/workflows/scratch/dev-build.yaml` (+0 -0) 📝 `.gitignore` (+2 -0) 📝 `README.md` (+11 -0) 📝 `cloud-deployments/aws/cloudformation/cloudformation_create_anythingllm.json` (+297 -184) 📝 `cloud-deployments/gcp/deployment/gcp_deploy_anything_llm.yaml` (+3 -3) ➕ `cloud-deployments/helm/charts/anythingllm/.helmignore` (+23 -0) ➕ `cloud-deployments/helm/charts/anythingllm/Chart.yaml` (+7 -0) ➕ `cloud-deployments/helm/charts/anythingllm/README.md` (+149 -0) ➕ `cloud-deployments/helm/charts/anythingllm/README.md.gotmpl` (+103 -0) ➕ `cloud-deployments/helm/charts/anythingllm/templates/NOTES.txt` (+28 -0) ➕ `cloud-deployments/helm/charts/anythingllm/templates/_helpers.tpl` (+62 -0) ➕ `cloud-deployments/helm/charts/anythingllm/templates/configmap.yaml` (+10 -0) ➕ `cloud-deployments/helm/charts/anythingllm/templates/deployment.yaml` (+83 -0) ➕ `cloud-deployments/helm/charts/anythingllm/templates/extra-objects.yaml` (+4 -0) ➕ `cloud-deployments/helm/charts/anythingllm/templates/ingress.yaml` (+61 -0) ➕ `cloud-deployments/helm/charts/anythingllm/templates/pvc.yaml` (+33 -0) _...and 44 more files_ </details> ### 📄 Description ## Overview This PR introduces an enhanced document processing pipeline that improves embedding quality through contextual enrichment and external OCR integration capabilities. ## Key Features ### Document Enrichment - Implements a one-time enrichment process that adds contextual metadata to document chunks during embedding - Enriches chunks with position information and document context to improve retrieval accuracy - Configurable enrichment settings via system configuration ### External OCR Integration - Adds API support for ingesting OCR data from external sources - Supports structured OCR field data with flexible field mapping - Maintains backward compatibility with existing document processing workflows ### Enhanced Error Handling - Improves error handling and validation throughout the vectorization pipeline - Adds comprehensive error logging for better debugging - Fixes cache logic bug in Zilliz provider ### TextSplitter Enhancements - Extends metadata support to include additional document fields - Maintains existing functionality while adding new capabilities ## Technical Details ### New Modules - `server/utils/documentEnrichment.js` - Core enrichment logic - `server/utils/ocrFieldParser.js` - OCR field parsing utilities ### Modified Components - Document upload endpoints now accept external OCR data - Zilliz vector provider enhanced with enrichment pipeline integration - TextSplitter extended with additional metadata fields - Improved error handling across document processing flow ## API Changes - `POST /v1/document/upload` and `POST /v1/document/upload/:folderName` now accept optional `externalOCRFields` parameter ## Testing - Verified with existing document processing workflows - Tested error handling and edge cases - Validated backward compatibility ## Notes - This enhancement is designed to work seamlessly with existing document processing - No breaking changes to existing APIs - Enrichment can be configured or disabled via system settings --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
yindo added the pull-request label 2026-02-22 18:36:23 -05:00
yindo closed this issue 2026-02-22 18:36:23 -05:00
yindo changed title from [PR #4692] # Document Enrichment Pipeline Enhancement to [PR #4692] [CLOSED] # Document Enrichment Pipeline Enhancement 2026-06-05 15:20:01 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#4718
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?