run-llama/create-llama
1
0
Fork 0
mirror of https://github.com/run-llama/create-llama.git synced 2026-07-16 11:04:26 -04:00
Code Issues 39 Packages Projects Releases 10 Wiki Activity

[PR #74] [MERGED] Use ingestion pipeline #227

New Issue
Closed
opened 2026-02-15 19:16:48 -05:00 by yindo · 0 comments
yindo commented 2026-02-15 19:16:48 -05:00
Owner
Copy Link
Copy Source

📋 Pull Request Information

Original PR: https://github.com/run-llama/create-llama/pull/74
Author: @leehuwuj
Created: 5/3/2024
Status: ✅ Merged
Merged: 5/14/2024
Merged by: @marcusschiesser

Base: main ← Head: lee/use-ingestion-pipeline

📝 Commits (6)

  • 02f360e use ingestion pipeline for vector stores
  • 2429140 add embedding dim for pg vector
  • 93d6496 remove redundant constants file
  • 42e980f add comments, use storage config from env
  • 8d25625 chore: Refactor file.py and generate.py for better code organization
  • da9ca2c Refactor file.py and generate.py for better code organization

📊 Changes

34 files changed (+241 additions, -371 deletions)

View changed files

📝 helpers/env-variables.ts (+2 -2)
📝 templates/components/loaders/python/file.py (+28 -8)
➖ templates/components/vectordbs/python/astra/__init__.py (+0 -0)
➖ templates/components/vectordbs/python/astra/generate.py (+0 -37)
➖ templates/components/vectordbs/python/astra/index.py (+0 -21)
➕ templates/components/vectordbs/python/astra/vectordb.py (+20 -0)
➖ templates/components/vectordbs/python/milvus/__init__.py (+0 -0)
➖ templates/components/vectordbs/python/milvus/generate.py (+0 -39)
➖ templates/components/vectordbs/python/milvus/index.py (+0 -22)
➕ templates/components/vectordbs/python/milvus/vectordb.py (+20 -0)
➖ templates/components/vectordbs/python/mongo/__init__.py (+0 -0)
➖ templates/components/vectordbs/python/mongo/generate.py (+0 -43)
➖ templates/components/vectordbs/python/mongo/index.py (+0 -20)
➕ templates/components/vectordbs/python/mongo/vectordb.py (+20 -0)
➖ templates/components/vectordbs/python/none/constants.py (+0 -1)
📝 templates/components/vectordbs/python/none/generate.py (+4 -3)
📝 templates/components/vectordbs/python/none/index.py (+5 -5)
➖ templates/components/vectordbs/python/pg/__init__.py (+0 -0)
➖ templates/components/vectordbs/python/pg/constants.py (+0 -2)
➖ templates/components/vectordbs/python/pg/generate.py (+0 -35)

...and 14 more files

📄 Description

This PR is not ready to merge because we're having issue with llama_index ingestion pipeline.
Experiment notebook: https://drive.google.com/file/d/1b19t5n6jtfex2UuKCRgxzNaCzk8a1ady/view?usp=sharing

Summary by CodeRabbit

  • New Features

    • Introduced logging functionality for enhanced monitoring.
    • Added functions for generating and persisting data indexes.
    • Implemented dynamic storage directory setting via environment variables.

  • Improvements

    • Enhanced exception handling for empty data directories.
    • Improved data ingestion pipeline and storage persistence mechanisms.

  • Refactor

    • Updated storage directory logic to use environment variables.
    • Reorganized functions for better modularity and clarity.

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/run-llama/create-llama/pull/74 **Author:** [@leehuwuj](https://github.com/leehuwuj) **Created:** 5/3/2024 **Status:** ✅ Merged **Merged:** 5/14/2024 **Merged by:** [@marcusschiesser](https://github.com/marcusschiesser) **Base:** `main` ← **Head:** `lee/use-ingestion-pipeline` --- ### 📝 Commits (6) - [`02f360e`](https://github.com/run-llama/create-llama/commit/02f360e8b4c5710748bd33be04657250e2f8ab25) use ingestion pipeline for vector stores - [`2429140`](https://github.com/run-llama/create-llama/commit/2429140cbc5ed0ac3b81767e643a79401f6fc58e) add embedding dim for pg vector - [`93d6496`](https://github.com/run-llama/create-llama/commit/93d649686fe5654aef6b88179de01435eade10e3) remove redundant constants file - [`42e980f`](https://github.com/run-llama/create-llama/commit/42e980fe61f039feda990be42df3bd33540e4479) add comments, use storage config from env - [`8d25625`](https://github.com/run-llama/create-llama/commit/8d2562597067a4cbe33da1ed11fb670d47e009ae) chore: Refactor file.py and generate.py for better code organization - [`da9ca2c`](https://github.com/run-llama/create-llama/commit/da9ca2cfc966449ad322d9828b6ceda5192019fb) Refactor file.py and generate.py for better code organization ### 📊 Changes **34 files changed** (+241 additions, -371 deletions) <details> <summary>View changed files</summary> 📝 `helpers/env-variables.ts` (+2 -2) 📝 `templates/components/loaders/python/file.py` (+28 -8) ➖ `templates/components/vectordbs/python/astra/__init__.py` (+0 -0) ➖ `templates/components/vectordbs/python/astra/generate.py` (+0 -37) ➖ `templates/components/vectordbs/python/astra/index.py` (+0 -21) ➕ `templates/components/vectordbs/python/astra/vectordb.py` (+20 -0) ➖ `templates/components/vectordbs/python/milvus/__init__.py` (+0 -0) ➖ `templates/components/vectordbs/python/milvus/generate.py` (+0 -39) ➖ `templates/components/vectordbs/python/milvus/index.py` (+0 -22) ➕ `templates/components/vectordbs/python/milvus/vectordb.py` (+20 -0) ➖ `templates/components/vectordbs/python/mongo/__init__.py` (+0 -0) ➖ `templates/components/vectordbs/python/mongo/generate.py` (+0 -43) ➖ `templates/components/vectordbs/python/mongo/index.py` (+0 -20) ➕ `templates/components/vectordbs/python/mongo/vectordb.py` (+20 -0) ➖ `templates/components/vectordbs/python/none/constants.py` (+0 -1) 📝 `templates/components/vectordbs/python/none/generate.py` (+4 -3) 📝 `templates/components/vectordbs/python/none/index.py` (+5 -5) ➖ `templates/components/vectordbs/python/pg/__init__.py` (+0 -0) ➖ `templates/components/vectordbs/python/pg/constants.py` (+0 -2) ➖ `templates/components/vectordbs/python/pg/generate.py` (+0 -35) _...and 14 more files_ </details> ### 📄 Description This PR is not ready to merge because we're having issue with llama_index ingestion pipeline. Experiment notebook: https://drive.google.com/file/d/1b19t5n6jtfex2UuKCRgxzNaCzk8a1ady/view?usp=sharing <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Introduced logging functionality for enhanced monitoring. - Added functions for generating and persisting data indexes. - Implemented dynamic storage directory setting via environment variables. - **Improvements** - Enhanced exception handling for empty data directories. - Improved data ingestion pipeline and storage persistence mechanisms. - **Refactor** - Updated storage directory logic to use environment variables. - Reorganized functions for better modularity and clarity. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
yindo added the pull-request label 2026-02-15 19:16:48 -05:00
yindo closed this issue 2026-02-15 19:16:48 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
pull-request
Mirrored from GitHub Pull Request
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: run-llama/create-llama#227
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?