llamaindex@0.0.41

Feat: add postgres vectordb (#308 )
* feat: integrate create-llama with postgresql * fix: get data for verification before inserting * feat: show available vector DBs based on framework
2026-07-13 22:17:48 -04:00 · 2024-01-08 15:18:24 +07:00 · 2024-01-05 14:13:52 +08:00 · 2024-01-04 18:03:00 -06:00 · 2024-01-04 10:19:18 +07:00 · 2024-01-04 10:19:18 +07:00
92 changed files with 49488 additions and 58398 deletions
@@ -36,7 +36,7 @@ jobs:
        working-directory: ./packages/create-llama
      - name: Run Playwright tests
        run: pnpm exec playwright test
-        env: 
+        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        working-directory: ./packages/create-llama
      - uses: actions/upload-artifact@v3
@@ -17,9 +17,11 @@ jobs:
      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
-          node-version-file: '.nvmrc'
+          node-version-file: ".nvmrc"
          cache: "pnpm"
      - name: Install dependencies
        run: pnpm install
      - name: Run lint
        run: pnpm run lint
+      - name: Run Prettier
+        run: pnpm run format
@@ -12,7 +12,7 @@ jobs:
      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
-          node-version-file: '.nvmrc'
+          node-version-file: ".nvmrc"
          cache: "pnpm"
      - name: Install dependencies
        run: pnpm install
@@ -1,5 +1,6 @@
 #!/usr/bin/env sh
 . "$(dirname -- "$0")/_/husky.sh"

+pnpm format
 pnpm lint
 npx lint-staged
@@ -0,0 +1,3 @@
+apps/docs/i18n
+pnpm-lock.yaml
+
@@ -8,7 +8,7 @@ Right now there are two packages of importance:

 packages/core which is the main NPM library llamaindex

-apps/simple is where the demo code lives
+examples is where the demo code lives

 ### Turborepo docs

@@ -47,7 +47,7 @@ We use Jest https://jestjs.io/ to write our test cases. Jest comes with a bunch

 ### Demo applications

-There is an existing ["simple"](/apps/simple/README.md) demos folder with mainly NodeJS scripts. Feel free to add additional demos to that folder. If you would like to try out your changes in the core package with a new demo, you need to run the build command in the README.
+There is an existing ["example"](/examples/README.md) demos folder with mainly NodeJS scripts. Feel free to add additional demos to that folder. If you would like to try out your changes in the core package with a new demo, you need to run the build command in the README.

 You can create new demo applications in the apps folder. Just run pnpm init in the folder after you create it to create its own package.json

@@ -56,7 +56,7 @@ You can create new demo applications in the apps folder. Just run pnpm init in t
 To install packages for a specific package or demo application, run

 ```
-pnpm add [NPM Package] --filter [package or application i.e. core or simple]
+pnpm add [NPM Package] --filter [package or application i.e. core or docs]
 ```

 To install packages for every package or application run
@@ -37,7 +37,7 @@ For more complex applications, our lower-level APIs allow advanced users to cust

 `npm install llamaindex`

-Our documentation includes [Installation Instructions](./installation.md) and a [Starter Tutorial](./starter.md) to build your first application.
+Our documentation includes [Installation Instructions](./installation.mdx) and a [Starter Tutorial](./starter.md) to build your first application.

 Once you're up and running, [High-Level Concepts](./concepts.md) has an overview of LlamaIndex's modular architecture. For more hands-on practical examples, look through our [End-to-End Tutorials](./end_to_end.md).

@@ -30,21 +30,21 @@ const config = {
  i18n: {
    defaultLocale: "en",
    locales: [
-    "en", 
-    "zh-Hans", 
-    "es",
-    "fr",
-    "de",
-    "ja",
-    "ko",
-    "pt",
-    "ar",
-    "it",
-    "tr",
-    "pl",
-    "nl",
-    "vi",
-    "th",
+      "en",
+      "zh-Hans",
+      "es",
+      "fr",
+      "de",
+      "ja",
+      "ko",
+      "pt",
+      "ar",
+      "it",
+      "tr",
+      "pl",
+      "nl",
+      "vi",
+      "th",
    ], // "fa", "ru", "ro", "sv", "hu", "cs", "el", "da", "fi", "he", "no", "hi", "in", "sl", "se", "sk", "uk", "bg", "hr", "lt", "lv", "et", "cat"
  },

@@ -66,10 +66,10 @@ const config = {
        },
        blog: false,
        gtag: {
-          trackingID: 'G-NB9B8LW9W5',
+          trackingID: "G-NB9B8LW9W5",
          anonymizeIP: true,
        },
-    }),
+      }),
    ],
  ],

@@ -39,7 +39,7 @@ LlamaIndex.TS هو إطار بيانات لتطبيقات LLM لاستيعاب

 `npm install llamaindex`

-تتضمن وثائقنا [تعليمات التثبيت](./installation.md) و[دليل البداية](./starter.md) لبناء تطبيقك الأول.
+تتضمن وثائقنا [تعليمات التثبيت](./installation.mdx) و[دليل البداية](./starter.md) لبناء تطبيقك الأول.

 بمجرد أن تكون جاهزًا وتعمل ، يحتوي [مفاهيم عالية المستوى](./concepts.md) على نظرة عامة على الهندسة المعمارية المتعددة المستويات لـ LlamaIndex. لمزيد من الأمثلة العملية التفصيلية ، يمكنك الاطلاع على [دروس النهاية إلى النهاية](./end_to_end.md).

@@ -41,7 +41,7 @@ LlamaIndex.TS предоставя основен набор от инструм

 `npm install llamaindex`

-Документацията ни включва [Инструкции за инсталиране](./installation.md) и [Урок за начинаещи](./starter.md), за да построите първото си приложение.
+Документацията ни включва [Инструкции за инсталиране](./installation.mdx) и [Урок за начинаещи](./starter.md), за да построите първото си приложение.

 След като сте готови, [Високо ниво концепции](./concepts.md) представя общ преглед на модулната архитектура на LlamaIndex. За повече практически примери, разгледайте нашите [Уроци от начало до край](./end_to_end.md).

@@ -39,7 +39,7 @@ Per a aplicacions més complexes, les nostres API de nivell inferior permeten al

 `npm install llamaindex`

-La nostra documentació inclou [Instruccions d'Instal·lació](./installation.md) i un [Tutorial d'Inici](./starter.md) per a construir la vostra primera aplicació.
+La nostra documentació inclou [Instruccions d'Instal·lació](./installation.mdx) i un [Tutorial d'Inici](./starter.md) per a construir la vostra primera aplicació.

 Un cop tingueu tot a punt, [Conceptes de Nivell Alt](./concepts.md) ofereix una visió general de l'arquitectura modular de LlamaIndex. Per a més exemples pràctics, consulteu els nostres [Tutorials de Principi a Fi](./end_to_end.md).

@@ -39,7 +39,7 @@ Pro složitější aplikace naše API na nižší úrovni umožňuje pokročilý

 `npm install llamaindex`

-Naše dokumentace obsahuje [Návod k instalaci](./installation.md) a [Úvodní tutoriál](./starter.md) pro vytvoření vaší první aplikace.
+Naše dokumentace obsahuje [Návod k instalaci](./installation.mdx) a [Úvodní tutoriál](./starter.md) pro vytvoření vaší první aplikace.

 Jakmile jste připraveni, [Vysokoúrovňové koncepty](./concepts.md) poskytují přehled o modulární architektuře LlamaIndexu. Pro více praktických příkladů se podívejte na naše [Tutoriály od začátku do konce](./end_to_end.md).

@@ -39,7 +39,7 @@ Til mere komplekse applikationer giver vores API'er på lavere niveau avancerede

 `npm install llamaindex`

-Vores dokumentation inkluderer [Installationsinstruktioner](./installation.md) og en [Starter Tutorial](./starter.md) til at bygge din første applikation.
+Vores dokumentation inkluderer [Installationsinstruktioner](./installation.mdx) og en [Starter Tutorial](./starter.md) til at bygge din første applikation.

 Når du er i gang, giver [Højniveaukoncepter](./concepts.md) et overblik over LlamaIndex's modulære arkitektur. For flere praktiske eksempler, kan du kigge igennem vores [End-to-End Tutorials](./end_to_end.md).

@@ -39,7 +39,7 @@ Für komplexere Anwendungen ermöglichen unsere APIs auf niedrigerer Ebene fortg

 `npm install llamaindex`

-Unsere Dokumentation enthält [Installationsanweisungen](./installation.md) und ein [Einführungstutorial](./starter.md), um Ihre erste Anwendung zu erstellen.
+Unsere Dokumentation enthält [Installationsanweisungen](./installation.mdx) und ein [Einführungstutorial](./starter.md), um Ihre erste Anwendung zu erstellen.

 Sobald Sie bereit sind, bietet [High-Level-Konzepte](./concepts.md) einen Überblick über die modulare Architektur von LlamaIndex. Für praktische Beispiele schauen Sie sich unsere [End-to-End-Tutorials](./end_to_end.md) an.

@@ -41,7 +41,7 @@ slug: /

 `npm install llamaindex`

-Η τεκμηρίωσή μας περιλαμβάνει [Οδηγίες Εγκατάστασης](./installation.md) και ένα [Εισαγωγικό Εκπαιδευτικό Πρόγραμμα](./starter.md) για να δημιουργήσετε την πρώτη σας εφαρμογή.
+Η τεκμηρίωσή μας περιλαμβάνει [Οδηγίες Εγκατάστασης](./installation.mdx) και ένα [Εισαγωγικό Εκπαιδευτικό Πρόγραμμα](./starter.md) για να δημιουργήσετε την πρώτη σας εφαρμογή.

 Αφού ξεκινήσετε, οι [Υψηλού Επιπέδου Έννοιες](./concepts.md) παρέχουν μια επισκόπηση της μοντουλαρισμένης αρχιτεκτονικής του LlamaIndex. Για περισσότερα πρακτικά παραδείγματα, ρίξτε μια ματιά στα [Ολοκληρωμένα Εκπαιδευτικά Προγράμματα](./end_to_end.md).

@@ -39,7 +39,7 @@ Para aplicaciones más complejas, nuestras API de nivel inferior permiten a los

 `npm install llamaindex`

-Nuestra documentación incluye [Instrucciones de instalación](./installation.md) y un [Tutorial de inicio](./starter.md) para construir tu primera aplicación.
+Nuestra documentación incluye [Instrucciones de instalación](./installation.mdx) y un [Tutorial de inicio](./starter.md) para construir tu primera aplicación.

 Una vez que estés en funcionamiento, [Conceptos de alto nivel](./concepts.md) ofrece una visión general de la arquitectura modular de LlamaIndex. Para obtener ejemplos prácticos más detallados, consulta nuestros [Tutoriales de extremo a extremo](./end_to_end.md).

@@ -39,7 +39,7 @@ Täpsemate rakenduste jaoks võimaldavad meie madalama taseme API-d edasijõudnu

 `npm install llamaindex`

-Meie dokumentatsioonis on [paigaldusjuhised](./installation.md) ja [algõpetus](./starter.md) oma esimese rakenduse loomiseks.
+Meie dokumentatsioonis on [paigaldusjuhised](./installation.mdx) ja [algõpetus](./starter.md) oma esimese rakenduse loomiseks.

 Kui olete valmis ja töötate, siis [kõrgtasemel kontseptsioonid](./concepts.md) annavad ülevaate LlamaIndexi moodularhitektuurist. Praktiliste näidete jaoks vaadake läbi meie [otsast lõpuni õpetused](./end_to_end.md).

@@ -41,7 +41,7 @@ API سطح بالای ما به کاربران مبتدی امکان استفا

 `npm install llamaindex`

-مستندات ما شامل [دستورالعمل نصب](./installation.md) و [آموزش شروع کار](./starter.md) برای ساخت اولین برنامه شما است.
+مستندات ما شامل [دستورالعمل نصب](./installation.mdx) و [آموزش شروع کار](./starter.md) برای ساخت اولین برنامه شما است.

 با راه اندازی و اجرا شدن، [مفاهیم سطح بالا](./concepts.md) یک نمای کلی از معماری ماژولار لاماایندکس را ارائه می دهد. برای مثال های عملی بیشتر، به [آموزش های پایان به پایان](./end_to_end.md) مراجعه کنید.

@@ -41,7 +41,7 @@ Monimutkaisempiin sovelluksiin tarjoamme matalamman tason API:t, jotka mahdollis

 `npm install llamaindex`

-Dokumentaatiostamme löydät [asennusohjeet](./installation.md) ja [aloitusopetusohjelman](./starter.md) ensimmäisen sovelluksesi rakentamiseen.
+Dokumentaatiostamme löydät [asennusohjeet](./installation.mdx) ja [aloitusopetusohjelman](./starter.md) ensimmäisen sovelluksesi rakentamiseen.

 Kun olet päässyt vauhtiin, [Korkean tason käsitteet](./concepts.md) antaa yleiskuvan LlamaIndexin modulaarisesta arkkitehtuurista. Lisää käytännön esimerkkejä löydät [Päästä päähän -opetusohjelmista](./end_to_end.md).

@@ -41,7 +41,7 @@ API הרמה הגבוהה שלנו מאפשר למשתמשים מתחילים ל

 `npm install llamaindex`

-התיעוד שלנו כולל [הוראות התקנה](./installation.md) ו[מדריך התחלה](./starter.md) לבניית היישום הראשון שלך.
+התיעוד שלנו כולל [הוראות התקנה](./installation.mdx) ו[מדריך התחלה](./starter.md) לבניית היישום הראשון שלך.

 כאשר אתה מוכן ורץ, [מושגים ברמה גבוהה](./concepts.md) מציג סקירה על ארכיטקטורה מודולרית של LlamaIndex. לדוגמאות פרקטיות יותר, עיין ב[מדריכים מתקדמים מתחילה ועד סוף](./end_to_end.md).

@@ -41,7 +41,7 @@ LlamaIndex.TS जावास्क्रिप्ट और TypeScript के

 `npm install llamaindex`

-हमारी दस्तावेज़ी में [स्थापना निर्देश](./installation.md) और [स्टार्टर ट्यूटोरियल](./starter.md) शामिल हैं, जिनका उपयोग करके आप अपना पहला एप्लिकेशन बना सकते हैं।
+हमारी दस्तावेज़ी में [स्थापना निर्देश](./installation.mdx) और [स्टार्टर ट्यूटोरियल](./starter.md) शामिल हैं, जिनका उपयोग करके आप अपना पहला एप्लिकेशन बना सकते हैं।

 एक बार जब आप शुरू हो जाएं, [उच्च स्तरीय अवधारणाएँ](./concepts.md) में LlamaIndex की मॉड्यूलर आर्किटेक्चर का अवलोकन है। अधिक हैंड्स-ऑन प्रैक्टिकल उदाहरणों के लिए, हमारे [एंड-टू-एंड ट्यूटोरियल](./end_to_end.md) को देखें।

@@ -39,7 +39,7 @@ Za složenije aplikacije, naše API-je niže razine omogućuju naprednim korisni

 `npm install llamaindex`

-Naša dokumentacija uključuje [Upute za instalaciju](./installation.md) i [Uvodni vodič](./starter.md) za izgradnju vaše prve aplikacije.
+Naša dokumentacija uključuje [Upute za instalaciju](./installation.mdx) i [Uvodni vodič](./starter.md) za izgradnju vaše prve aplikacije.

 Kada ste spremni za rad, [Visokorazinski koncepti](./concepts.md) pružaju pregled modularne arhitekture LlamaIndex-a. Za praktične primjere, pogledajte naše [Vodiče od početka do kraja](./end_to_end.md).

@@ -39,7 +39,7 @@ A komplexebb alkalmazásokhoz a mélyebb szintű API-k lehetővé teszik a halad

 `npm install llamaindex`

-A dokumentációnk tartalmazza a [Telepítési utasításokat](./installation.md) és egy [Kezdő útmutatót](./starter.md) az első alkalmazás létrehozásához.
+A dokumentációnk tartalmazza a [Telepítési utasításokat](./installation.mdx) és egy [Kezdő útmutatót](./starter.md) az első alkalmazás létrehozásához.

 Miután elindultál, a [Magas szintű fogalmak](./concepts.md) áttekintést ad a LlamaIndex moduláris architektúrájáról. További gyakorlati példákért tekintsd meg az [End-to-End útmutatóinkat](./end_to_end.md).

@@ -39,7 +39,7 @@ Untuk aplikasi yang lebih kompleks, API tingkat lebih rendah kami memungkinkan p

 `npm install llamaindex`

-Dokumentasi kami mencakup [Instruksi Instalasi](./installation.md) dan [Tutorial Awal](./starter.md) untuk membangun aplikasi pertama Anda.
+Dokumentasi kami mencakup [Instruksi Instalasi](./installation.mdx) dan [Tutorial Awal](./starter.md) untuk membangun aplikasi pertama Anda.

 Setelah Anda mulai, [Konsep Tingkat Tinggi](./concepts.md) memberikan gambaran tentang arsitektur modular LlamaIndex. Untuk contoh praktis yang lebih mendalam, lihat [Tutorial End-to-End](./end_to_end.md).

@@ -39,7 +39,7 @@ Per applicazioni più complesse, le nostre API di livello inferiore consentono a

 `npm install llamaindex`

-La nostra documentazione include le [Istruzioni di installazione](./installation.md) e un [Tutorial introduttivo](./starter.md) per creare la tua prima applicazione.
+La nostra documentazione include le [Istruzioni di installazione](./installation.mdx) e un [Tutorial introduttivo](./starter.md) per creare la tua prima applicazione.

 Una volta che sei pronto, i [Concetti di alto livello](./concepts.md) offrono una panoramica dell'architettura modulare di LlamaIndex. Per ulteriori esempi pratici, consulta i nostri [Tutorial end-to-end](./end_to_end.md).

@@ -39,7 +39,7 @@ LlamaIndex.TSは、JavaScriptとTypeScriptを使用してLLMアプリを構築

 `npm install llamaindex`

-私たちのドキュメントには、[インストール手順](./installation.md)と[スターターチュートリアル](./starter.md)が含まれており、最初のアプリケーションの構築をサポートします。
+私たちのドキュメントには、[インストール手順](./installation.mdx)と[スターターチュートリアル](./starter.md)が含まれており、最初のアプリケーションの構築をサポートします。

 一度準備ができたら、[ハイレベルなコンセプト](./concepts.md)では、LlamaIndexのモジュラーアーキテクチャの概要を説明しています。より実践的な例については、[エンドツーエンドのチュートリアル](./end_to_end.md)を参照してください。

@@ -39,7 +39,7 @@ LlamaIndex.TS는 JavaScript와 TypeScript로 LLM 앱을 개발하는 모든 사

 `npm install llamaindex`

-저희 문서에는 [설치 지침](./installation.md)과 [스타터 튜토리얼](./starter.md)이 포함되어 있어 첫 번째 애플리케이션을 빌드할 수 있습니다.
+저희 문서에는 [설치 지침](./installation.mdx)과 [스타터 튜토리얼](./starter.md)이 포함되어 있어 첫 번째 애플리케이션을 빌드할 수 있습니다.

 한 번 시작하면, [고수준 개념](./concepts.md)에서 LlamaIndex의 모듈식 아키텍처에 대한 개요를 확인할 수 있습니다. 더 많은 실전 예제를 원하신다면, [End-to-End 튜토리얼](./end_to_end.md)을 참조해주세요.

@@ -39,7 +39,7 @@ Sudėtingesnėms programoms mūsų žemesnio lygio API leidžia pažengusiems na

 `npm install llamaindex`

-Mūsų dokumentacija apima [įdiegimo instrukcijas](./installation.md) ir [pradžios vadovą](./starter.md), skirtą sukurti pirmąją aplikaciją.
+Mūsų dokumentacija apima [įdiegimo instrukcijas](./installation.mdx) ir [pradžios vadovą](./starter.md), skirtą sukurti pirmąją aplikaciją.

 Kai jau esate paleidę, [aukšto lygio konceptai](./concepts.md) pateikia apžvalgą apie LlamaIndex modularią architektūrą. Norėdami gauti daugiau praktinių pavyzdžių, peržiūrėkite mūsų [nuo pradžių iki pabaigos vadovus](./end_to_end.md).

@@ -41,7 +41,7 @@ Lielākām un sarežģītākām lietojumprogrammām mūsu zemāka līmeņa API

 `npm install llamaindex`

-Mūsu dokumentācijā ir iekļautas [Instalācijas instrukcijas](./installation.md) un [Sākuma pamācība](./starter.md), lai izveidotu savu pirmo lietojumprogrammu.
+Mūsu dokumentācijā ir iekļautas [Instalācijas instrukcijas](./installation.mdx) un [Sākuma pamācība](./starter.md), lai izveidotu savu pirmo lietojumprogrammu.

 Kad esat gatavs, [Augsta līmeņa koncepti](./concepts.md) sniedz pārskatu par LlamaIndex modulāro arhitektūru. Lai iegūtu vairāk praktisku piemēru, apskatiet mūsu [Galēji līdz galam pamācības](./end_to_end.md).

@@ -39,7 +39,7 @@ Voor complexere toepassingen stellen onze API's op lager niveau gevorderde gebru

 `npm install llamaindex`

-Onze documentatie bevat [Installatie-instructies](./installation.md) en een [Starterzelfstudie](./starter.md) om uw eerste toepassing te bouwen.
+Onze documentatie bevat [Installatie-instructies](./installation.mdx) en een [Starterzelfstudie](./starter.md) om uw eerste toepassing te bouwen.

 Zodra u aan de slag bent, geeft [Hoog-niveau Concepten](./concepts.md) een overzicht van de modulaire architectuur van LlamaIndex. Voor meer praktische voorbeelden kunt u onze [End-to-End Tutorials](./end_to_end.md) bekijken.

@@ -39,7 +39,7 @@ For mer komplekse applikasjoner lar våre lavnivå-APIer avanserte brukere tilpa

 `npm install llamaindex`

-Dokumentasjonen vår inkluderer [Installasjonsinstruksjoner](./installation.md) og en [Starterveiledning](./starter.md) for å bygge din første applikasjon.
+Dokumentasjonen vår inkluderer [Installasjonsinstruksjoner](./installation.mdx) og en [Starterveiledning](./starter.md) for å bygge din første applikasjon.

 Når du er oppe og kjører, gir [Høynivåkonsepter](./concepts.md) en oversikt over LlamaIndex sin modulære arkitektur. For mer praktiske eksempler, kan du se gjennom våre [End-to-End veiledninger](./end_to_end.md).

@@ -39,7 +39,7 @@ Dla bardziej zaawansowanych aplikacji nasze API na niższym poziomie umożliwia

 `npm install llamaindex`

-Nasza dokumentacja zawiera [Instrukcje instalacji](./installation.md) oraz [Samouczek dla początkujących](./starter.md), który pomoże Ci zbudować swoją pierwszą aplikację.
+Nasza dokumentacja zawiera [Instrukcje instalacji](./installation.mdx) oraz [Samouczek dla początkujących](./starter.md), który pomoże Ci zbudować swoją pierwszą aplikację.

 Gdy już będziesz gotowy, [Wysokopoziomowe koncepcje](./concepts.md) zawierają przegląd modułowej architektury LlamaIndex. Jeśli chcesz zobaczyć praktyczne przykłady, zapoznaj się z naszymi [Samouczkami od początku do końca](./end_to_end.md).

@@ -39,7 +39,7 @@ Para aplicativos mais complexos, nossas APIs de nível inferior permitem que usu

 `npm install llamaindex`

-Nossa documentação inclui [Instruções de Instalação](./installation.md) e um [Tutorial Inicial](./starter.md) para construir seu primeiro aplicativo.
+Nossa documentação inclui [Instruções de Instalação](./installation.mdx) e um [Tutorial Inicial](./starter.md) para construir seu primeiro aplicativo.

 Depois de estar pronto para começar, [Conceitos de Alto Nível](./concepts.md) oferece uma visão geral da arquitetura modular do LlamaIndex. Para exemplos práticos mais detalhados, consulte nossos [Tutoriais de Ponta a Ponta](./end_to_end.md).

@@ -39,7 +39,7 @@ Pentru aplicații mai complexe, API-urile noastre de nivel inferior permit utili

 `npm install llamaindex`

-Documentația noastră include [Instrucțiuni de instalare](./installation.md) și un [Tutorial de pornire](./starter.md) pentru a construi prima ta aplicație.
+Documentația noastră include [Instrucțiuni de instalare](./installation.mdx) și un [Tutorial de pornire](./starter.md) pentru a construi prima ta aplicație.

 Odată ce ai început, [Concepte de nivel înalt](./concepts.md) oferă o prezentare generală a arhitecturii modulare a LlamaIndex. Pentru mai multe exemple practice, consultă [Tutorialele de la cap la coadă](./end_to_end.md).

@@ -39,7 +39,7 @@ LlamaIndex.TS предоставляет основной набор инстр

 `npm install llamaindex`

-Наша документация включает [Инструкции по установке](./installation.md) и [Стартовое руководство](./starter.md) для создания вашего первого приложения.
+Наша документация включает [Инструкции по установке](./installation.mdx) и [Стартовое руководство](./starter.md) для создания вашего первого приложения.

 Когда вы начнете работу, [Высокоуровневые концепции](./concepts.md) предоставляют обзор модульной архитектуры LlamaIndex. Для более практических примеров руководство [Полный цикл руководств](./end_to_end.md) будет полезно.

@@ -39,7 +39,7 @@ Za složenije aplikacije, naše API-je na nižem nivou omogućavaju naprednim ko

 `npm install llamaindex`

-Naša dokumentacija uključuje [Uputstva za instalaciju](./installation.md) i [Uvodni tutorijal](./starter.md) za izgradnju vaše prve aplikacije.
+Naša dokumentacija uključuje [Uputstva za instalaciju](./installation.mdx) i [Uvodni tutorijal](./starter.md) za izgradnju vaše prve aplikacije.

 Kada ste spremni za rad, [Koncepti na visokom nivou](./concepts.md) pružaju pregled modularne arhitekture LlamaIndex-a. Za praktične primere, pogledajte naše [Vodiče od početka do kraja](./end_to_end.md).

@@ -39,7 +39,7 @@ Za bolj kompleksne aplikacije naša nizkonivojska API omogoča naprednim uporabn

 `npm install llamaindex`

-Naša dokumentacija vključuje [Navodila za namestitev](./installation.md) in [Vodič za začetek](./starter.md), ki vam pomagata zgraditi vašo prvo aplikacijo.
+Naša dokumentacija vključuje [Navodila za namestitev](./installation.mdx) in [Vodič za začetek](./starter.md), ki vam pomagata zgraditi vašo prvo aplikacijo.

 Ko ste pripravljeni, [Visokonivojski koncepti](./concepts.md) ponujajo pregled modularne arhitekture LlamaIndex-a. Za več praktičnih primerov si oglejte naše [Vodiče od začetka do konca](./end_to_end.md).

@@ -39,7 +39,7 @@ Pre zložitejšie aplikácie naša nižšia úroveň API umožňuje pokročilým

 `npm install llamaindex`

-Naša dokumentácia obsahuje [Inštalačné pokyny](./installation.md) a [Úvodný tutoriál](./starter.md) pre vytvorenie vašej prvej aplikácie.
+Naša dokumentácia obsahuje [Inštalačné pokyny](./installation.mdx) a [Úvodný tutoriál](./starter.md) pre vytvorenie vašej prvej aplikácie.

 Keď už máte všetko pripravené, [Vysokoúrovňové koncepty](./concepts.md) poskytujú prehľad o modulárnej architektúre LlamaIndexu. Pre viac praktických príkladov si prečítajte naše [Tutoriály od začiatku do konca](./end_to_end.md).

@@ -39,7 +39,7 @@ För mer komplexa applikationer tillåter våra lägre nivå-API:er avancerade a

 `npm install llamaindex`

-Vår dokumentation inkluderar [Installationsinstruktioner](./installation.md) och en [Starterhandledning](./starter.md) för att bygga din första applikation.
+Vår dokumentation inkluderar [Installationsinstruktioner](./installation.mdx) och en [Starterhandledning](./starter.md) för att bygga din första applikation.

 När du är igång, ger [Högnivåkoncept](./concepts.md) en översikt över LlamaIndex modulära arkitektur. För mer praktiska exempel, titta igenom våra [Steg-för-steg handledningar](./end_to_end.md).

@@ -39,7 +39,7 @@ API ระดับสูงของเราช่วยให้ผู้ใ

 `npm install llamaindex`

-เอกสารของเราประกอบด้วย[คำแนะนำการติดตั้ง](./installation.md)และ[บทแนะนำเบื้องต้น](./starter.md)เพื่อสร้างแอปพลิเคชันครั้งแรกของคุณ
+เอกสารของเราประกอบด้วย[คำแนะนำการติดตั้ง](./installation.mdx)และ[บทแนะนำเบื้องต้น](./starter.md)เพื่อสร้างแอปพลิเคชันครั้งแรกของคุณ

 เมื่อคุณเริ่มใช้งานแล้ว [แนวคิดระดับสูง](./concepts.md) มีภาพรวมของสถาปัตยกรรมแบบโมดูลของ LlamaIndex สำหรับตัวอย่างที่เป็นปฏิบัติจริงมากขึ้น โปรดดูที่ [บทแนะนำจบสู่จบ](./end_to_end.md) เพื่อตัวอย่างที่ใช้งานได้จริง

@@ -41,7 +41,7 @@ Daha karmaşık uygulamalar için, düşük seviyeli API'larımız, gelişmiş k

 `npm install llamaindex`

-Dökümantasyonumuz, [Kurulum Talimatları](./installation.md) ve ilk uygulamanızı oluşturmanız için bir [Başlangıç Kılavuzu](./starter.md) içerir.
+Dökümantasyonumuz, [Kurulum Talimatları](./installation.mdx) ve ilk uygulamanızı oluşturmanız için bir [Başlangıç Kılavuzu](./starter.md) içerir.

 Çalışmaya başladıktan sonra, [Yüksek Düzeyli Kavramlar](./concepts.md) LlamaIndex'in modüler mimarisinin bir genel bakışını sunar. Daha fazla pratik örnek için [Uçtan Uca Öğreticilerimize](./end_to_end.md) göz atabilirsiniz.

@@ -39,7 +39,7 @@ LlamaIndex.TS надає основний набір інструментів,

 `npm install llamaindex`

-Наша документація містить [Інструкції з встановлення](./installation.md) та [Посібник для початківців](./starter.md) для створення вашої першої програми.
+Наша документація містить [Інструкції з встановлення](./installation.mdx) та [Посібник для початківців](./starter.md) для створення вашої першої програми.

 Після того, як ви розпочнете роботу, [Високорівневі концепції](./concepts.md) містить огляд модульної архітектури LlamaIndex. Для більш практичних прикладів роботи, перегляньте наші [Посібники з кінця в кінець](./end_to_end.md).

@@ -39,7 +39,7 @@ API cấp cao của chúng tôi cho phép người dùng mới bắt đầu sử

 `npm install llamaindex`

-Tài liệu của chúng tôi bao gồm [Hướng dẫn cài đặt](./installation.md) và [Hướng dẫn bắt đầu](./starter.md) để xây dựng ứng dụng đầu tiên của bạn.
+Tài liệu của chúng tôi bao gồm [Hướng dẫn cài đặt](./installation.mdx) và [Hướng dẫn bắt đầu](./starter.md) để xây dựng ứng dụng đầu tiên của bạn.

 Khi bạn đã sẵn sàng, [Khái niệm cấp cao](./concepts.md) cung cấp một cái nhìn tổng quan về kiến trúc mô-đun của LlamaIndex. Để có thêm ví dụ thực tế, hãy xem qua [Hướng dẫn từ đầu đến cuối](./end_to_end.md).

@@ -37,7 +37,7 @@ LlamaIndex.TS 提供了一套核心工具，对于任何使用JavaScript和TypeS

 `npm install llamaindex`

-我们的文档包括[安装说明](./installation.md)和一个[入门教程](./starter.md)，帮助你构建第一个应用程序。
+我们的文档包括[安装说明](./installation.mdx)和一个[入门教程](./starter.md)，帮助你构建第一个应用程序。

 一旦你开始运行，[高级概念](./concepts.md)有一个LlamaIndex模块化架构的概览。更多实践例子，请浏览我们的[端到端教程](./end_to_end.md)。

@@ -39,7 +39,7 @@ LlamaIndex.TS 提供了一組核心工具，對於使用 JavaScript 和 TypeScri

 `npm install llamaindex`

-我們的文檔包括[安裝說明](./installation.md)和[入門教程](./starter.md)，以構建您的第一個應用程序。
+我們的文檔包括[安裝說明](./installation.mdx)和[入門教程](./starter.md)，以構建您的第一個應用程序。

 一旦您開始運行，[高級概念](./concepts.md)提供了 LlamaIndex 模塊化架構的概述。如果需要更多實際的操作示例，請查看我們的[端到端教程](./end_to_end.md)。

@@ -1,15 +1,16 @@
 import { program } from "commander";
-import { AudioTranscriptReader, TranscribeParams, VectorStoreIndex } from "llamaindex";
+import {
+  AudioTranscriptReader,
+  TranscribeParams,
+  VectorStoreIndex,
+} from "llamaindex";
 import { stdin as input, stdout as output } from "node:process";
 // readline/promises is still experimental so not in @types/node yet
 // @ts-ignore
 import readline from "node:readline/promises";

 program
-  .option(
-    "-a, --audio [string]",
-    "URL or path of the audio file to transcribe",
-  )
+  .option("-a, --audio [string]", "URL or path of the audio file to transcribe")
  .option("-i, --transcript-id [string]", "ID of the AssemblyAI transcript")
  .action(async (options) => {
    if (!process.env.ASSEMBLYAI_API_KEY) {
@@ -26,9 +27,7 @@ program
    } else if (options.transcriptId) {
      params = options.transcriptId;
    } else {
-      console.log(
-        "You must provide either an --audio or a --transcript-id",
-      );
+      console.log("You must provide either an --audio or a --transcript-id");
      return;
    }

@@ -1,29 +1,25 @@
+import { Document, SimpleMongoReader, VectorStoreIndex } from "llamaindex";
 import { MongoClient } from "mongodb";
-import { Document } from "../../packages/core/src/Node";
-import { VectorStoreIndex } from "../../packages/core/src/indices";
-import { SimpleMongoReader } from "../../packages/core/src/readers/SimpleMongoReader";

 import { stdin as input, stdout as output } from "node:process";
 import readline from "node:readline/promises";

 async function main() {
  //Dummy test code
-  const query: object = { _id: "waldo" };
-  const options: object = {};
-  const projections: object = { embedding: 0 };
+  const filterQuery = {};
  const limit: number = Infinity;
-  const uri: string = process.env.MONGODB_URI ?? "fake_uri";
+  const uri: string = process.env.MONGODB_URI ?? "mongodb://localhost:27017";
  const client: MongoClient = new MongoClient(uri);

  //Where the real code starts
  const MR = new SimpleMongoReader(client);
  const documents: Document[] = await MR.loadData(
-    "data",
-    "posts",
-    1,
-    {},
-    options,
-    projections,
+    "db",
+    "collection",
+    ["text"],
+    "",
+    filterQuery,
+    limit,
  );

  //
@@ -43,7 +39,7 @@ async function main() {
  // console.log(nodes);

  //
-  //Making Vector Store from documents
+  // Making Vector Store from documents
  //

  const index = await VectorStoreIndex.fromDocuments(documents);
@@ -45,6 +45,39 @@ async function loadAndIndex() {
  await client.close();
 }

-loadAndIndex();
+/**
+ * This method is document in https://www.mongodb.com/docs/atlas/atlas-search/create-index/#create-an-fts-index-programmatically
+ * But, while testing a 'CommandNotFound' error occurred, so we're not using this here.
+ */
+async function createSearchIndex() {
+  const client = new MongoClient(mongoUri);
+  const database = client.db(databaseName);
+  const collection = database.collection(vectorCollectionName);
+
+  // define your Atlas Search index
+  const index = {
+    name: indexName,
+    definition: {
+      /* search index definition fields */
+      mappings: {
+        dynamic: true,
+        fields: [
+          {
+            type: "vector",
+            path: "embedding",
+            numDimensions: 1536,
+            similarity: "cosine",
+          },
+        ],
+      },
+    },
+  };
+  // run the helper method
+  const result = await collection.createSearchIndex(index);
+  console.log("Successfully created search index:", result);
+  await client.close();
+}
+
+loadAndIndex().catch(console.error);

 // you can't query your index yet because you need to create a vector search index in mongodb's UI now
@@ -76,7 +76,7 @@ Now if all has gone well you should be able to log in to the Mongo Atlas UI and

 Now it's time to create the vector search index so that you can query the data.
 It's not yet possible to programmatically create a vector search index using the [`createIndex`](https://www.mongodb.com/docs/manual/reference/method/db.collection.createIndex/) function, therefore we have to create one manually in the UI.
-To do so, first, click the Search tab, and then click "Create Search Index":
+To do so, first, click the 'Atlas Search' tab, and then click "Create Search Index":

 ![MongoDB Atlas create search index](./docs/4_search_tab.png)

@@ -88,16 +88,14 @@ Now under "database and collection" select `tiny_tweets_db` and within that sele

 ```json
 {
-  "mappings": {
-    "dynamic": true,
-    "fields": {
-      "embedding": {
-        "dimensions": 1536,
-        "similarity": "cosine",
-        "type": "knnVector"
-      }
+  "fields": [
+    {
+      "type": "vector",
+      "path": "embedding",
+      "numDimensions": 1536,
+      "similarity": "cosine"
    }
-  }
+  ]
 }
 ```

@@ -0,0 +1,39 @@
+import { Ollama } from "llamaindex";
+
+(async () => {
+  const llm = new Ollama({ model: "llama2", temperature: 0.75 });
+  {
+    const response = await llm.chat([
+      { content: "Tell me a joke.", role: "user" },
+    ]);
+    console.log("Response 1:", response.message.content);
+  }
+  {
+    const response = await llm.complete("How are you?");
+    console.log("Response 2:", response.message.content);
+  }
+  {
+    const response = await llm.chat(
+      [{ content: "Tell me a joke.", role: "user" }],
+      undefined,
+      true,
+    );
+    console.log("Response 3:");
+    for await (const message of response) {
+      process.stdout.write(message); // no newline
+    }
+    console.log(); // newline
+  }
+  {
+    const response = await llm.complete("How are you?", undefined, true);
+    console.log("Response 4:");
+    for await (const message of response) {
+      process.stdout.write(message); // no newline
+    }
+    console.log(); // newline
+  }
+  {
+    const embedding = await llm.getTextEmbedding("Hello world!");
+    console.log("Embedding:", embedding);
+  }
+})();
@@ -1,19 +1,36 @@
 # Postgres Vector Store

-There are two scripts available here: load-docs.ts and query.ts
+There are two scripts available here: `load-docs.ts` and `query.ts`

 ## Prerequisites

+### Start a DB Instance
+
 You'll need a postgres database instance against which to run these scripts. A simple docker command would look like this:

 > `docker run -d --rm --name vector-db -p 5432:5432 -e "POSTGRES_HOST_AUTH_METHOD=trust" ankane/pgvector`

-Set the PGHOST and PGUSER (and PGPASSWORD) environment variables to match your database setup.
-
-You'll also need a value for OPENAI_API_KEY in your environment.
-
 **NOTE:** Using `--rm` in the example docker command above means that the vector store will be deleted every time the container is stopped. For production purposes, use a volume to ensure persistence across restarts.

+If you prefer using a managed service, you can use [Timescale](https://docs.timescale.com/use-timescale/latest/services/create-a-service/?ref=timescale.com) to create a PostgreSQL database instance in the cloud as an alternative.
+
+### Set up Environment
+
+Having created a DB instance, you can then set up environment variables for your database connection:
+
+```bash
+export PGHOST=<your database host>
+export PGUSER=<your database user>
+export PGPASSWORD=<your database password>
+export PGDATABASE=<your database name>
+export PGPORT=<your database port>
+export OPENAI_API_KEY=<your openai api key>
+```
+
+Set the environment variables above to match your database setup.
+Note that you'll also need an OpenAI key (`OPENAI_API_KEY`) in your environment.
+You're now ready to start the scripts.
+
 ## Setup and Loading Docs

 Read and follow the instructions in the README.md file located one directory up to make sure your JS/TS dependencies are set up. The commands listed below are also run from that parent directory.
@@ -22,7 +39,7 @@ To import documents and save the embedding vectors to your database:

 > `npx ts-node pg-vector-store/load-docs.ts data`

-where data is the directory containing your input files. Using the _data_ directory in the example above will read all of the files in that directory using the llamaindexTS default readers for each file type.
+where data is the directory containing your input files. Using the `data` directory in the example above will read all of the files in that directory using the LlamaIndexTS default readers for each file type.

 ## RAG Querying

@@ -1,11 +1,11 @@
 // load-docs.ts
 import fs from "fs/promises";
 import {
+  PineconeVectorStore,
  SimpleDirectoryReader,
  storageContextFromDefaults,
  VectorStoreIndex,
 } from "llamaindex";
-import { PineconeVectorStore } from "llamaindex";

 async function getSourceFilenames(sourceDir: string) {
  return await fs
@@ -1,6 +1,8 @@
-import { VectorStoreIndex } from "llamaindex";
-import { serviceContextFromDefaults } from "llamaindex";
-import { PineconeVectorStore } from "llamaindex";
+import {
+  PineconeVectorStore,
+  VectorStoreIndex,
+  serviceContextFromDefaults,
+} from "llamaindex";

 async function main() {
  const readline = require("readline").createInterface({
@@ -0,0 +1,11 @@
+{
+  "compilerOptions": {
+    "target": "es2016",
+    "module": "commonjs",
+    "esModuleInterop": true,
+    "forceConsistentCasingInFileNames": true,
+    "strict": true,
+    "skipLibCheck": true
+  },
+  "include": ["./**/*.ts"]
+}
@@ -3,7 +3,8 @@
  "scripts": {
    "build": "turbo run build",
    "dev": "turbo run dev",
-    "format": "prettier --write \"**/*.{js,jsx,ts,tsx,md}\"",
+    "format": "prettier --ignore-unknown --cache --check .",
+    "format:write": "prettier --ignore-unknown --write .",
    "lint": "turbo run lint",
    "prepare": "husky install",
    "test": "turbo run test",
@@ -11,18 +12,18 @@
    "publish-snapshot": "turbo run build lint test --filter=\"!docs\" && changeset version --snapshot && changeset publish"
  },
  "devDependencies": {
-    "@changesets/cli": "^2.26.2",
-    "@turbo/gen": "^1.10.16",
-    "@types/jest": "^29.5.10",
-    "eslint": "^8.54.0",
+    "@changesets/cli": "^2.27.1",
+    "@turbo/gen": "^1.11.2",
+    "@types/jest": "^29.5.11",
+    "eslint": "^8.56.0",
    "eslint-config-custom": "workspace:*",
    "husky": "^8.0.3",
    "jest": "^29.7.0",
-    "lint-staged": "^15.1.0",
-    "prettier": "^3.1.0",
+    "lint-staged": "^15.2.0",
+    "prettier": "^3.1.1",
    "prettier-plugin-organize-imports": "^3.2.4",
    "ts-jest": "^29.1.1",
-    "turbo": "^1.10.16"
+    "turbo": "^1.11.2"
  },
  "packageManager": "pnpm@8.10.5+sha256.a4bd9bb7b48214bbfcd95f264bd75bb70d100e5d4b58808f5cd6ab40c6ac21c5",
  "pnpm": {
@@ -1,5 +1,14 @@
 # llamaindex

+## 0.0.41
+
+### Patch Changes
+
+- c835f78: Use compromise as sentence tokenizer
+- c835f78: Removed pdf-parse, and directly use latest pdf.js
+- c835f78: Added pinecone vector DB
+- c835f78: Added support for Ollama
+
 ## 0.0.40

 ### Patch Changes
@@ -1,6 +1,6 @@
 {
  "name": "llamaindex",
-  "version": "0.0.40",
+  "version": "0.0.41",
  "license": "MIT",
  "dependencies": {
    "@anthropic-ai/sdk": "^0.9.1",
@@ -11,7 +11,6 @@
    "@xenova/transformers": "^2.10.0",
    "assemblyai": "^4.0.0",
    "compromise": "^14.10.1",
-    "crypto-js": "^4.2.0",
    "file-type": "^18.7.0",
    "js-tiktoken": "^1.0.8",
    "lodash": "^4.17.21",
@@ -32,7 +31,6 @@
    "wink-nlp": "^1.14.3"
  },
  "devDependencies": {
-    "@types/crypto-js": "^4.2.1",
    "@types/jest": "^29.5.11",
    "@types/lodash": "^4.14.202",
    "@types/node": "^18.19.2",
@@ -1,4 +1,5 @@
-import CryptoJS from "crypto-js";
+import _ from "lodash";
+import { createHash } from "node:crypto";
 import path from "path";
 import { v4 as uuidv4 } from "uuid";

@@ -141,12 +142,26 @@ export abstract class BaseNode<T extends Metadata = Metadata> {
  }

  /**
-   * Used with built in JSON.stringify
-   * @returns
+   * Called by built in JSON.stringify (see https://javascript.info/json)
+   * Properties are read-only as they are not deep-cloned (not necessary for stringification).
+   * @see toMutableJSON - use to return a mutable JSON instead
   */
  toJSON(): Record<string, any> {
    return { ...this, type: this.getType() };
  }
+
+  clone(): BaseNode {
+    return jsonToNode(this.toMutableJSON()) as BaseNode;
+  }
+
+  /**
+   * Converts the object to a JSON representation.
+   * Properties can be safely modified as a deep clone of the properties are created.
+   * @return {Record<string, any>} - The JSON representation of the object.
+   */
+  toMutableJSON(): Record<string, any> {
+    return _.cloneDeep(this.toJSON());
+  }
 }

 /**
@@ -177,13 +192,13 @@ export class TextNode<T extends Metadata = Metadata> extends BaseNode<T> {
   * @returns
   */
  generateHash() {
-    const hashFunction = CryptoJS.algo.SHA256.create();
+    const hashFunction = createHash("sha256");
    hashFunction.update(`type=${this.getType()}`);
    hashFunction.update(
      `startCharIdx=${this.startCharIdx} endCharIdx=${this.endCharIdx}`,
    );
    hashFunction.update(this.getContent(MetadataMode.ALL));
-    return hashFunction.finalize().toString(CryptoJS.enc.Base64);
+    return hashFunction.digest("base64");
  }

  getType(): ObjectType {
@@ -1,5 +1,5 @@
-import nlp from 'compromise'
-import { EOL } from 'node:os'
+import nlp from "compromise";
+import { EOL } from "node:os";
 // GitHub translated
 import { globalsHelper } from "./GlobalsHelper";
 import { DEFAULT_CHUNK_OVERLAP, DEFAULT_CHUNK_SIZE } from "./constants";
@@ -20,11 +20,15 @@ class TextSplit {
 type SplitRep = { text: string; numTokens: number };

 export const defaultSentenceTokenizer = (text: string): string[] => {
-  return nlp(text).sentences().json().map((sentence: any) => sentence.text);
+  return nlp(text)
+    .sentences()
+    .json()
+    .map((sentence: any) => sentence.text);
 };

 // Refs: https://github.com/fxsjy/jieba/issues/575#issuecomment-359637511
-const resentencesp = /([﹒﹔﹖﹗．；。！？]["’”」』]{0,2}|：(?=["‘“「『]{1,2}|$))/;
+const resentencesp =
+  /([﹒﹔﹖﹗．；。！？]["’”」』]{0,2}|：(?=["‘“「『]{1,2}|$))/;
 /**
 * Tokenizes sentences. Suitable for Chinese, Japanese, and Korean. Use instead of `defaultSentenceTokenizer`.
 * @param text
@@ -46,7 +50,7 @@ export function cjkSentenceTokenizer(sentence: string): string[] {
  return slist.filter((s) => s.length > 0);
 }

-export const defaultParagraphSeparator = EOL + EOL + EOL
+export const defaultParagraphSeparator = EOL + EOL + EOL;

 // In theory there's also Mac style \r only, but it's pre-OSX and I don't think
 // many documents will use it.
@@ -227,9 +231,10 @@ export class SentenceSplitter {
          // push curent doc list to docs
          docs.push(
            new TextSplit(
-              curChunkSentences.map((sentence) => sentence.text).
-                join(" ").
-                trim(),
+              curChunkSentences
+                .map((sentence) => sentence.text)
+                .join(" ")
+                .trim(),
            ),
          );
        }
@@ -9,7 +9,6 @@ export * from "./PromptHelper";
 export * from "./QueryEngine";
 export * from "./QuestionGenerator";
 export * from "./Response";
-export * from "./synthesizers";
 export * from "./Retriever";
 export * from "./ServiceContext";
 export * from "./TextSplitter";
@@ -29,3 +28,4 @@ export * from "./readers/SimpleDirectoryReader";
 export * from "./readers/SimpleMongoReader";
 export * from "./readers/base";
 export * from "./storage";
+export * from "./synthesizers";
@@ -4,7 +4,6 @@ import {
  ImageNode,
  MetadataMode,
  ObjectType,
-  jsonToNode,
  splitNodesByType,
 } from "../../Node";
 import { BaseQueryEngine, RetrieverQueryEngine } from "../../QueryEngine";
@@ -24,7 +23,7 @@ import {
 } from "../../storage/StorageContext";
 import { BaseIndexStore } from "../../storage/indexStore/types";
 import { VectorStore } from "../../storage/vectorStore/types";
-import { ResponseSynthesizer, BaseSynthesizer } from "../../synthesizers";
+import { BaseSynthesizer } from "../../synthesizers";
 import {
  BaseIndex,
  BaseIndexInit,
@@ -278,7 +277,7 @@ export class VectorStoreIndex extends BaseIndex<IndexDict> {
        type === ObjectType.INDEX ||
        type === ObjectType.IMAGE
      ) {
-        const nodeWithoutEmbedding = jsonToNode(nodes[i].toJSON());
+        const nodeWithoutEmbedding = nodes[i].clone();
        nodeWithoutEmbedding.embedding = undefined;
        this.indexStruct.addNode(nodeWithoutEmbedding, newIds[i]);
        this.docStore.addDocuments([nodeWithoutEmbedding], true);
@@ -10,6 +10,7 @@ import {

 import { ChatCompletionMessageParam } from "openai/resources";
 import { LLMOptions } from "portkey-ai";
+import { MessageContent } from "../ChatEngine";
 import { globalsHelper, Tokenizers } from "../GlobalsHelper";
 import {
  ANTHROPIC_AI_PROMPT,
@@ -27,7 +28,6 @@ import {
 import { getOpenAISession, OpenAISession } from "./openai";
 import { getPortkeySession, PortkeySession } from "./portkey";
 import { ReplicateSession } from "./replicate";
-import { MessageContent } from "../ChatEngine";

 export type MessageType =
  | "user"
@@ -1,2 +1,3 @@
 export * from "./LLM";
 export * from "./mistral";
+export { Ollama } from "./ollama";
@@ -0,0 +1,200 @@
+import { ok } from "node:assert";
+import { MessageContent } from "../ChatEngine";
+import { CallbackManager, Event } from "../callbacks/CallbackManager";
+import { BaseEmbedding } from "../embeddings";
+import { ChatMessage, ChatResponse, LLM, LLMMetadata } from "./LLM";
+
+const messageAccessor = (data: any) => data.message.content;
+const completionAccessor = (data: any) => data.response;
+
+// https://github.com/jmorganca/ollama
+export class Ollama extends BaseEmbedding implements LLM {
+  readonly hasStreaming = true;
+
+  // https://ollama.ai/library
+  model: string;
+  baseURL: string = "http://127.0.0.1:11434";
+  temperature: number = 0.7;
+  topP: number = 0.9;
+  contextWindow: number = 4096;
+  requestTimeout: number = 60 * 1000; // Default is 60 seconds
+  additionalChatOptions?: Record<string, unknown>;
+  callbackManager?: CallbackManager;
+
+  constructor(
+    init: Partial<Ollama> & {
+      // model is required
+      model: string;
+    },
+  ) {
+    super();
+    this.model = init.model;
+    Object.assign(this, init);
+  }
+
+  get metadata(): LLMMetadata {
+    return {
+      model: this.model,
+      temperature: this.temperature,
+      topP: this.topP,
+      maxTokens: undefined,
+      contextWindow: this.contextWindow,
+      tokenizer: undefined,
+    };
+  }
+
+  async chat<
+    T extends boolean | undefined = undefined,
+    R = T extends true ? AsyncGenerator<string, void, unknown> : ChatResponse,
+  >(
+    messages: ChatMessage[],
+    parentEvent?: Event | undefined,
+    streaming?: T,
+  ): Promise<R> {
+    const payload = {
+      model: this.model,
+      messages: messages.map((message) => ({
+        role: message.role,
+        content: message.content,
+      })),
+      stream: !!streaming,
+      options: {
+        temperature: this.temperature,
+        num_ctx: this.contextWindow,
+        top_p: this.topP,
+        ...this.additionalChatOptions,
+      },
+    };
+    const response = await fetch(`${this.baseURL}/api/chat`, {
+      body: JSON.stringify(payload),
+      method: "POST",
+      signal: AbortSignal.timeout(this.requestTimeout),
+      headers: {
+        "Content-Type": "application/json",
+      },
+    });
+    if (!streaming) {
+      const raw = await response.json();
+      const { message } = raw;
+      return {
+        message: {
+          role: "assistant",
+          content: message.content,
+        },
+        raw,
+      } satisfies ChatResponse as R;
+    } else {
+      const stream = response.body;
+      ok(stream, "stream is null");
+      ok(stream instanceof ReadableStream, "stream is not readable");
+      return this.streamChat(stream, messageAccessor, parentEvent) as R;
+    }
+  }
+
+  private async *streamChat(
+    stream: ReadableStream<Uint8Array>,
+    accessor: (data: any) => string,
+    parentEvent?: Event,
+  ): AsyncGenerator<string, void, unknown> {
+    const reader = stream.getReader();
+    while (true) {
+      const { done, value } = await reader.read();
+      if (done) {
+        return;
+      }
+      const lines = Buffer.from(value)
+        .toString("utf-8")
+        .split("\n")
+        .map((line) => line.trim());
+      for (const line of lines) {
+        if (line === "") {
+          continue;
+        }
+        const json = JSON.parse(line);
+        if (json.error) {
+          throw new Error(json.error);
+        }
+        yield accessor(json);
+      }
+    }
+  }
+
+  async complete<
+    T extends boolean | undefined = undefined,
+    R = T extends true ? AsyncGenerator<string, void, unknown> : ChatResponse,
+  >(
+    prompt: MessageContent,
+    parentEvent?: Event | undefined,
+    streaming?: T | undefined,
+  ): Promise<R> {
+    const payload = {
+      model: this.model,
+      prompt: prompt,
+      stream: !!streaming,
+      options: {
+        temperature: this.temperature,
+        num_ctx: this.contextWindow,
+        top_p: this.topP,
+        ...this.additionalChatOptions,
+      },
+    };
+    const response = await fetch(`${this.baseURL}/api/generate`, {
+      body: JSON.stringify(payload),
+      method: "POST",
+      signal: AbortSignal.timeout(this.requestTimeout),
+      headers: {
+        "Content-Type": "application/json",
+      },
+    });
+    if (!streaming) {
+      const raw = await response.json();
+      return {
+        message: {
+          role: "assistant",
+          content: raw.response,
+        },
+        raw,
+      } satisfies ChatResponse as R;
+    } else {
+      const stream = response.body;
+      ok(stream, "stream is null");
+      ok(stream instanceof ReadableStream, "stream is not readable");
+      return this.streamChat(stream, completionAccessor, parentEvent) as R;
+    }
+  }
+
+  tokens(messages: ChatMessage[]): number {
+    throw new Error("Method not implemented.");
+  }
+
+  private async getEmbedding(prompt: string): Promise<number[]> {
+    const payload = {
+      model: this.model,
+      prompt,
+      options: {
+        temperature: this.temperature,
+        num_ctx: this.contextWindow,
+        top_p: this.topP,
+        ...this.additionalChatOptions,
+      },
+    };
+    const response = await fetch(`${this.baseURL}/api/embeddings`, {
+      body: JSON.stringify(payload),
+      method: "POST",
+      signal: AbortSignal.timeout(this.requestTimeout),
+      headers: {
+        "Content-Type": "application/json",
+      },
+    });
+    const { embedding } = await response.json();
+    return embedding;
+  }
+
+  async getTextEmbedding(text: string): Promise<number[]> {
+    return this.getEmbedding(text);
+  }
+
+  async getQueryEmbedding(query: string): Promise<number[]> {
+    return this.getEmbedding(query);
+  }
+}
@@ -1,7 +1,7 @@
 import mammoth from "mammoth";
 import { Document } from "../Node";
-import { DEFAULT_FS } from "../storage/constants";
 import { GenericFileSystem } from "../storage/FileSystem";
+import { DEFAULT_FS } from "../storage/constants";
 import { BaseReader } from "./base";

 export class DocxReader implements BaseReader {
@@ -9,7 +9,7 @@ export { SimpleKVStore } from "./kvStore/SimpleKVStore";
 export * from "./kvStore/types";
 export { AstraDBVectorStore } from "./vectorStore/AstraDBVectorStore";
 export { MongoDBAtlasVectorSearch } from "./vectorStore/MongoDBAtlasVectorStore";
+export { PGVectorStore } from "./vectorStore/PGVectorStore";
 export { PineconeVectorStore } from "./vectorStore/PineconeVectorStore";
 export { SimpleVectorStore } from "./vectorStore/SimpleVectorStore";
-export { PGVectorStore } from "./vectorStore/PGVectorStore";
 export * from "./vectorStore/types";
@@ -11,33 +11,43 @@ export const PGVECTOR_TABLE = "llamaindex_embedding";

 /**
 * Provides support for writing and querying vector data in Postgres.
+ * Note: Can't be used with data created using the Python version of the vector store (https://docs.llamaindex.ai/en/stable/examples/vector_stores/postgres.html)
 */
 export class PGVectorStore implements VectorStore {
  storesText: boolean = true;

  private collection: string = "";
+  private schemaName: string = PGVECTOR_SCHEMA;
+  private tableName: string = PGVECTOR_TABLE;
+  private connectionString: string | undefined = undefined;

-  /*
-    FROM pg LIBRARY:
-    type Config = {
-      user?: string, // default process.env.PGUSER || process.env.USER
-      password?: string or function, //default process.env.PGPASSWORD
-      host?: string, // default process.env.PGHOST
-      database?: string, // default process.env.PGDATABASE || user
-      port?: number, // default process.env.PGPORT
-      connectionString?: string, // e.g. postgres://user:password@host:5432/database
-      ssl?: any, // passed directly to node.TLSSocket, supports all tls.connect options
-      types?: any, // custom type parsers
-      statement_timeout?: number, // number of milliseconds before a statement in query will time out, default is no timeout
-      query_timeout?: number, // number of milliseconds before a query call will timeout, default is no timeout
-      application_name?: string, // The name of the application that created this Client instance
-      connectionTimeoutMillis?: number, // number of milliseconds to wait for connection, default is no timeout
-      idle_in_transaction_session_timeout?: number // number of milliseconds before terminating any session with an open idle transaction, default is no timeout
-    }  
-  */
-  db?: pg.Client;
+  private db?: pg.Client;

-  constructor() {}
+  /**
+   * Constructs a new instance of the PGVectorStore
+   *
+   * If the `connectionString` is not provided the following env variables are
+   * used to connect to the DB:
+   * PGHOST=<your database host>
+   * PGUSER=<your database user>
+   * PGPASSWORD=<your database password>
+   * PGDATABASE=<your database name>
+   * PGPORT=<your database port>
+   *
+   * @param {object} config - The configuration settings for the instance.
+   * @param {string} config.schemaName - The name of the schema (optional). Defaults to PGVECTOR_SCHEMA.
+   * @param {string} config.tableName - The name of the table (optional). Defaults to PGVECTOR_TABLE.
+   * @param {string} config.connectionString - The connection string (optional).
+   */
+  constructor(config?: {
+    schemaName?: string;
+    tableName?: string;
+    connectionString?: string;
+  }) {
+    this.schemaName = config?.schemaName ?? PGVECTOR_SCHEMA;
+    this.tableName = config?.tableName ?? PGVECTOR_TABLE;
+    this.connectionString = config?.connectionString;
+  }

  /**
   * Setter for the collection property.
@@ -66,7 +76,9 @@ export class PGVectorStore implements VectorStore {
      try {
        // Create DB connection
        // Read connection params from env - see comment block above
-        const db = new pg.Client();
+        const db = new pg.Client({
+          connectionString: this.connectionString,
+        });
        await db.connect();

        // Check vector extension
@@ -88,9 +100,9 @@ export class PGVectorStore implements VectorStore {
  }

  private async checkSchema(db: pg.Client) {
-    await db.query(`CREATE SCHEMA IF NOT EXISTS ${PGVECTOR_SCHEMA}`);
+    await db.query(`CREATE SCHEMA IF NOT EXISTS ${this.schemaName}`);

-    const tbl = `CREATE TABLE IF NOT EXISTS ${PGVECTOR_SCHEMA}.${PGVECTOR_TABLE}(
+    const tbl = `CREATE TABLE IF NOT EXISTS ${this.schemaName}.${this.tableName}(
      id uuid DEFAULT gen_random_uuid() PRIMARY KEY,
      external_id VARCHAR,
      collection VARCHAR,
@@ -100,16 +112,14 @@ export class PGVectorStore implements VectorStore {
    )`;
    await db.query(tbl);

-    const idxs = `CREATE INDEX IF NOT EXISTS idx_${PGVECTOR_TABLE}_external_id ON ${PGVECTOR_SCHEMA}.${PGVECTOR_TABLE} (external_id);
-      CREATE INDEX IF NOT EXISTS idx_${PGVECTOR_TABLE}_collection ON ${PGVECTOR_SCHEMA}.${PGVECTOR_TABLE} (collection);`;
+    const idxs = `CREATE INDEX IF NOT EXISTS idx_${this.tableName}_external_id ON ${this.schemaName}.${this.tableName} (external_id);
+      CREATE INDEX IF NOT EXISTS idx_${this.tableName}_collection ON ${this.schemaName}.${this.tableName} (collection);`;
    await db.query(idxs);

    // TODO add IVFFlat or HNSW indexing?
    return db;
  }

-  // isEmbeddingQuery?: boolean | undefined;
-
  /**
   * Connects to the database specified in environment vars.
   * This method also checks and creates the vector extension,
@@ -126,7 +136,7 @@ export class PGVectorStore implements VectorStore {
   * @returns The result of the delete query.
   */
  async clearCollection() {
-    const sql: string = `DELETE FROM ${PGVECTOR_SCHEMA}.${PGVECTOR_TABLE} 
+    const sql: string = `DELETE FROM ${this.schemaName}.${this.tableName} 
      WHERE collection = $1`;

    const db = (await this.getDb()) as pg.Client;
@@ -135,25 +145,8 @@ export class PGVectorStore implements VectorStore {
    return ret;
  }

-  /**
-   * Adds vector record(s) to the table.
-   * NOTE: Uses the collection property controlled by setCollection/getCollection.
-   * @param embeddingResults The Nodes to be inserted, optionally including metadata tuples.
-   * @returns A list of zero or more id values for the created records.
-   */
-  async add(embeddingResults: BaseNode<Metadata>[]): Promise<string[]> {
-    if (embeddingResults.length == 0) {
-      console.debug("Empty list sent to PGVectorStore::add");
-      return Promise.resolve([]);
-    }
-
-    const sql: string = `INSERT INTO ${PGVECTOR_SCHEMA}.${PGVECTOR_TABLE} 
-      (id, external_id, collection, document, metadata, embeddings) 
-      VALUES ($1, $2, $3, $4, $5, $6)`;
-
-    const db = (await this.getDb()) as pg.Client;
-
-    let ret: string[] = [];
+  private getDataToInsert(embeddingResults: BaseNode<Metadata>[]) {
+    const result = [];
    for (let index = 0; index < embeddingResults.length; index++) {
      const row = embeddingResults[index];

@@ -170,11 +163,37 @@ export class PGVectorStore implements VectorStore {
        "[" + row.getEmbedding().join(",") + "]",
      ];

+      result.push(params);
+    }
+    return result;
+  }
+
+  /**
+   * Adds vector record(s) to the table.
+   * NOTE: Uses the collection property controlled by setCollection/getCollection.
+   * @param embeddingResults The Nodes to be inserted, optionally including metadata tuples.
+   * @returns A list of zero or more id values for the created records.
+   */
+  async add(embeddingResults: BaseNode<Metadata>[]): Promise<string[]> {
+    if (embeddingResults.length == 0) {
+      console.debug("Empty list sent to PGVectorStore::add");
+      return Promise.resolve([]);
+    }
+
+    const sql: string = `INSERT INTO ${this.schemaName}.${this.tableName} 
+      (id, external_id, collection, document, metadata, embeddings) 
+      VALUES ($1, $2, $3, $4, $5, $6)`;
+
+    const db = (await this.getDb()) as pg.Client;
+    const data = this.getDataToInsert(embeddingResults);
+
+    let ret: string[] = [];
+    for (let index = 0; index < data.length; index++) {
+      const params = data[index];
      try {
        const result = await db.query(sql, params);
-
        if (result.rows.length) {
-          id = result.rows[0].id as string;
+          const id = result.rows[0].id as string;
          ret.push(id);
        }
      } catch (err) {
@@ -197,7 +216,7 @@ export class PGVectorStore implements VectorStore {
    const collectionCriteria = this.collection.length
      ? "AND collection = $2"
      : "";
-    const sql: string = `DELETE FROM ${PGVECTOR_SCHEMA}.${PGVECTOR_TABLE} 
+    const sql: string = `DELETE FROM ${this.schemaName}.${this.tableName} 
      WHERE id = $1 ${collectionCriteria}`;

    const db = (await this.getDb()) as pg.Client;
@@ -230,7 +249,7 @@ export class PGVectorStore implements VectorStore {
    const sql = `SELECT 
        v.*, 
        embeddings <-> $1 s 
-      FROM ${PGVECTOR_SCHEMA}.${PGVECTOR_TABLE} v
+      FROM ${this.schemaName}.${this.tableName} v
      ${where}
      ORDER BY s 
      LIMIT ${max}
@@ -1,8 +1,10 @@
-import { VectorStore, 
-        VectorStoreQuery, 
-        VectorStoreQueryResult,
-        ExactMatchFilter, 
-        MetadataFilters } from "./types";
+import {
+  ExactMatchFilter,
+  MetadataFilters,
+  VectorStore,
+  VectorStoreQuery,
+  VectorStoreQueryResult,
+} from "./types";

 import { BaseNode, Document, Metadata, MetadataMode } from "../../Node";
 import { GenericFileSystem } from "../FileSystem";
@@ -142,7 +144,7 @@ export class PineconeVectorStore implements VectorStore {
      topK: query.similarityTopK,
      include_values: true,
      include_metadara: true,
-      filter: filter
+      filter: filter,
    };

    const idx = await this.index();
@@ -16,20 +16,17 @@ export function nodeToMetadata(
  textField: string = DEFAULT_TEXT_KEY,
  flatMetadata: boolean = false,
 ): Metadata {
-  const nodeObj = node.toJSON();
-  const metadata = node.metadata;
+  const { metadata, embedding, ...rest } = node.toMutableJSON();

  if (flatMetadata) {
-    validateIsFlat(node.metadata);
+    validateIsFlat(metadata);
  }

  if (removeText) {
-    nodeObj[textField] = "";
+    rest[textField] = "";
  }

-  nodeObj["embedding"] = null;
-
-  metadata["_node_content"] = JSON.stringify(nodeObj);
+  metadata["_node_content"] = JSON.stringify(rest);
  metadata["_node_type"] = node.constructor.name.replace("_", ""); // remove leading underscore to be compatible with Python

  metadata["document_id"] = node.sourceNode?.nodeId || "None";
@@ -40,17 +37,24 @@ export function nodeToMetadata(
 }

 export function metadataDictToNode(metadata: Metadata): BaseNode {
-  const nodeContent = metadata["_node_content"];
+  const {
+    _node_content: nodeContent,
+    _node_type: nodeType,
+    document_id,
+    doc_id,
+    ref_doc_id,
+    ...rest
+  } = metadata;
  if (!nodeContent) {
    throw new Error("Node content not found in metadata.");
  }
  const nodeObj = JSON.parse(nodeContent);
+  nodeObj.metadata = rest;

  // Note: we're using the name of the class stored in `_node_type`
  // and not the type attribute to reconstruct
  // the node. This way we're compatible with LlamaIndex Python
-  const node_type = metadata["_node_type"];
-  switch (node_type) {
+  switch (nodeType) {
    case "IndexNode":
      return jsonToNode(nodeObj, ObjectType.INDEX);
    default:
@@ -1,3 +1,5 @@
+import { Document } from "../Node";
+import { ServiceContext, serviceContextFromDefaults } from "../ServiceContext";
 import {
  CallbackManager,
  RetrievalCallbackResponse,
@@ -7,12 +9,7 @@ import { OpenAIEmbedding } from "../embeddings";
 import { SummaryIndex } from "../indices/summary";
 import { VectorStoreIndex } from "../indices/vectorStore/VectorStoreIndex";
 import { OpenAI } from "../llm/LLM";
-import { Document } from "../Node";
-import {
-  ResponseSynthesizer,
-  SimpleResponseBuilder,
-} from "../synthesizers";
-import { ServiceContext, serviceContextFromDefaults } from "../ServiceContext";
+import { ResponseSynthesizer, SimpleResponseBuilder } from "../synthesizers";
 import { mockEmbeddingModel, mockLlmGeneration } from "./utility/mockOpenAI";

 // Mock the OpenAI getOpenAISession function during testing
@@ -1,4 +1,4 @@
-import { cjkSentenceTokenizer, SentenceSplitter } from '../TextSplitter'
+import { cjkSentenceTokenizer, SentenceSplitter } from "../TextSplitter";

 describe("SentenceSplitter", () => {
  test("initializes", () => {
@@ -88,7 +88,9 @@ describe("SentenceSplitter", () => {
      chunkingTokenizerFn: cjkSentenceTokenizer,
    });

-    const splits = sentenceSplitter.splitText("此后如竟没有炬火：我便是唯一的光。倘若有了炬火，出了太阳，我们自然心悦诚服的消失。不但毫无不平，而且还要随喜赞美这炬火或太阳；因为他照了人类，连我都在内。");
+    const splits = sentenceSplitter.splitText(
+      "此后如竟没有炬火：我便是唯一的光。倘若有了炬火，出了太阳，我们自然心悦诚服的消失。不但毫无不平，而且还要随喜赞美这炬火或太阳；因为他照了人类，连我都在内。",
+    );
    expect(splits).toEqual([
      "此后如竟没有炬火：我便是唯一的光。",
      "倘若有了炬火，出了太阳，我们自然心悦诚服的消失。",
@@ -0,0 +1,47 @@
+import { Document, MetadataMode } from "../Node";
+import {
+  metadataDictToNode,
+  nodeToMetadata,
+} from "../storage/vectorStore/utils";
+
+describe("Testing VectorStore utils", () => {
+  let node: Document;
+
+  beforeEach(() => {
+    node = new Document({
+      text: "text",
+      metadata: { meta1: "Some metadata" },
+    });
+  });
+
+  test("nodeToMetadata should not modify a node's metadata", () => {
+    nodeToMetadata(node, true);
+    expect(node.metadata).toEqual({ meta1: "Some metadata" });
+  });
+  test("metadataDictToNode should reconstructs node and remove text (except embedding)", () => {
+    const metadata = nodeToMetadata(node, true);
+    const newNode = metadataDictToNode(metadata);
+    expect(newNode.metadata).toEqual({ meta1: "Some metadata" });
+    expect(() => newNode.getEmbedding()).toThrow();
+    expect(newNode.getContent(MetadataMode.NONE)).toEqual("");
+  });
+  test("metadataDictToNode should reconstructs node (except embedding)", () => {
+    const metadata = nodeToMetadata(node, false);
+    const newNode = metadataDictToNode(metadata);
+    expect(newNode.metadata).toEqual({ meta1: "Some metadata" });
+    expect(newNode.getContent(MetadataMode.NONE)).toEqual("text");
+    expect(() => newNode.getEmbedding()).toThrow();
+  });
+  test("metadataDictToNode should not allow deep metadata if flatMetadata is true", () => {
+    node.metadata = { meta: { meta: "meta" } };
+    expect(() => nodeToMetadata(node, false, "text", true)).toThrow();
+  });
+  test("metadataDictToNode should throw an error when node content not found in metadata", () => {
+    const faultyMetadata = {
+      _node_type: "IndexNode",
+    };
+    expect(() => {
+      metadataDictToNode(faultyMetadata);
+    }).toThrow();
+  });
+});
@@ -8,18 +8,10 @@
    "preserveWatchOutput": true,
    "skipLibCheck": true,
    "strict": true,
-    "lib": [
-      "es2015",
-      "dom"
-    ],
+    "lib": ["es2015", "dom"],
    "target": "ES2015",
    "resolveJsonModule": true,
-    "typeRoots": [
-      "./types",
-      "./node_modules/@types"
-    ]
+    "typeRoots": ["./types", "./node_modules/@types"]
  },
-  "exclude": [
-    "node_modules"
-  ],
-}
+  "exclude": ["node_modules"]
+}
@@ -58,4 +58,4 @@
  "engines": {
    "node": ">=16.14.0"
  }
-}
+}
@@ -1,9 +1,12 @@
 import ciInfo from "ci-info";
+import fs from "fs";
+import path from "path";
 import { blue, green } from "picocolors";
 import prompts from "prompts";
 import { InstallAppArgs } from "./create-app";
 import { COMMUNITY_OWNER, COMMUNITY_REPO } from "./helpers/constant";
 import { getRepoRootFolders } from "./helpers/repo";
+import { TemplateFramework } from "./templates";

 export type QuestionArgs = Omit<InstallAppArgs, "appPath" | "packageManager">;

@@ -26,6 +29,31 @@ const handlers = {
  },
 };

+const getVectorDbChoices = (framework: TemplateFramework) => {
+  const choices = [
+    {
+      title: "No, just store the data in the file system",
+      value: "none",
+    },
+    { title: "MongoDB", value: "mongo" },
+    { title: "PostgreSQL", value: "pg" },
+  ];
+
+  const vectodbLang = framework === "fastapi" ? "python" : "typescript";
+  const compPath = path.join(__dirname, "components");
+  const vectordbPath = path.join(compPath, "vectordbs", vectodbLang);
+
+  const availableChoices = fs
+    .readdirSync(vectordbPath)
+    .filter((file) => fs.statSync(path.join(vectordbPath, file)).isDirectory());
+
+  const displayedChoices = choices.filter((choice) =>
+    availableChoices.includes(choice.value),
+  );
+
+  return displayedChoices;
+};
+
 export const onPromptState = (state: any) => {
  if (state.aborted) {
    // If we don't re-enable the terminal cursor before exiting
@@ -233,30 +261,23 @@ export const askQuestions = async (
      program.engine = engine;
      preferences.engine = engine;
    }
-  }
-
-  if (program.engine !== "simple" && !program.vectorDb) {
-    if (ciInfo.isCI) {
-      program.vectorDb = getPrefOrDefault("vectorDb");
-    } else {
-      const { vectorDb } = await prompts(
-        {
-          type: "select",
-          name: "vectorDb",
-          message: "Would you like to use a vector database?",
-          choices: [
-            {
-              title: "No, just store the data in the file system",
-              value: "none",
-            },
-            { title: "MongoDB", value: "mongo" },
-          ],
-          initial: 0,
-        },
-        handlers,
-      );
-      program.vectorDb = vectorDb;
-      preferences.vectorDb = vectorDb;
+    if (program.engine !== "simple" && !program.vectorDb) {
+      if (ciInfo.isCI) {
+        program.vectorDb = getPrefOrDefault("vectorDb");
+      } else {
+        const { vectorDb } = await prompts(
+          {
+            type: "select",
+            name: "vectorDb",
+            message: "Would you like to use a vector database?",
+            choices: getVectorDbChoices(program.framework),
+            initial: 0,
+          },
+          handlers,
+        );
+        program.vectorDb = vectorDb;
+        preferences.vectorDb = vectorDb;
+      }
    }
  }

@@ -0,0 +1,45 @@
+/* eslint-disable turbo/no-undeclared-env-vars */
+import * as dotenv from "dotenv";
+import {
+  PGVectorStore,
+  SimpleDirectoryReader,
+  VectorStoreIndex,
+  storageContextFromDefaults,
+} from "llamaindex";
+import {
+  PGVECTOR_SCHEMA,
+  PGVECTOR_TABLE,
+  STORAGE_DIR,
+  checkRequiredEnvVars,
+} from "./shared.mjs";
+
+dotenv.config();
+
+async function loadAndIndex() {
+  // load objects from storage and convert them into LlamaIndex Document objects
+  const documents = await new SimpleDirectoryReader().loadData({
+    directoryPath: STORAGE_DIR,
+  });
+
+  // create postgres vector store
+  const vectorStore = new PGVectorStore({
+    connectionString: process.env.PG_CONNECTION_STRING,
+    schemaName: PGVECTOR_SCHEMA,
+    tableName: PGVECTOR_TABLE,
+  });
+  vectorStore.setCollection(STORAGE_DIR);
+  vectorStore.clearCollection();
+
+  // create index from all the Documents
+  console.log("Start creating embeddings...");
+  const storageContext = await storageContextFromDefaults({ vectorStore });
+  await VectorStoreIndex.fromDocuments(documents, { storageContext });
+  console.log(`Successfully created embeddings.`);
+}
+
+(async () => {
+  checkRequiredEnvVars();
+  await loadAndIndex();
+  console.log("Finished generating storage.");
+  process.exit(0);
+})();
@@ -0,0 +1,39 @@
+/* eslint-disable turbo/no-undeclared-env-vars */
+import {
+  ContextChatEngine,
+  LLM,
+  PGVectorStore,
+  VectorStoreIndex,
+  serviceContextFromDefaults,
+} from "llamaindex";
+import {
+  CHUNK_OVERLAP,
+  CHUNK_SIZE,
+  PGVECTOR_SCHEMA,
+  PGVECTOR_TABLE,
+  checkRequiredEnvVars,
+} from "./shared.mjs";
+
+async function getDataSource(llm: LLM) {
+  checkRequiredEnvVars();
+  const pgvs = new PGVectorStore({
+    connectionString: process.env.PG_CONNECTION_STRING,
+    schemaName: PGVECTOR_SCHEMA,
+    tableName: PGVECTOR_TABLE,
+  });
+  const serviceContext = serviceContextFromDefaults({
+    llm,
+    chunkSize: CHUNK_SIZE,
+    chunkOverlap: CHUNK_OVERLAP,
+  });
+  return await VectorStoreIndex.fromVectorStore(pgvs, serviceContext);
+}
+
+export async function createChatEngine(llm: LLM) {
+  const index = await getDataSource(llm);
+  const retriever = index.asRetriever({ similarityTopK: 5 });
+  return new ContextChatEngine({
+    chatModel: llm,
+    retriever,
+  });
+}
@@ -0,0 +1,24 @@
+export const STORAGE_DIR = "./data";
+export const CHUNK_SIZE = 512;
+export const CHUNK_OVERLAP = 20;
+export const PGVECTOR_SCHEMA = "public";
+export const PGVECTOR_TABLE = "llamaindex_embedding";
+
+const REQUIRED_ENV_VARS = ["PG_CONNECTION_STRING", "OPENAI_API_KEY"];
+
+export function checkRequiredEnvVars() {
+  const missingEnvVars = REQUIRED_ENV_VARS.filter((envVar) => {
+    return !process.env[envVar];
+  });
+
+  if (missingEnvVars.length > 0) {
+    console.log(
+      `The following environment variables are required but missing: ${missingEnvVars.join(
+        ", ",
+      )}`,
+    );
+    throw new Error(
+      `Missing environment variables: ${missingEnvVars.join(", ")}`,
+    );
+  }
+}
@@ -49,6 +49,11 @@ const createEnvLocalFile = async (
      content += `MONGODB_VECTOR_INDEX=\n`;
      break;
    }
+    case "pg": {
+      content += `# For generating a connection URI, see https://docs.timescale.com/use-timescale/latest/services/create-a-service\n`;
+      content += `PG_CONNECTION_STRING=\n`;
+      break;
+    }
  }

  if (content) {
@@ -4,7 +4,7 @@ export type TemplateType = "simple" | "streaming" | "community";
 export type TemplateFramework = "nextjs" | "express" | "fastapi";
 export type TemplateEngine = "simple" | "context";
 export type TemplateUI = "html" | "shadcn";
-export type TemplateVectorDB = "none" | "mongo";
+export type TemplateVectorDB = "none" | "mongo" | "pg";

 export interface InstallTemplateArgs {
  appName: string;
@@ -1,3 +1,3 @@
 {
  "extends": "eslint:recommended"
-}
+}
@@ -7,4 +7,4 @@
    "skipLibCheck": true,
    "moduleResolution": "node"
  }
-}
+}
@@ -1,3 +1,3 @@
 {
  "extends": "eslint:recommended"
-}
+}
@@ -7,4 +7,4 @@
    "skipLibCheck": true,
    "moduleResolution": "node"
  }
-}
+}
@@ -1,11 +1,7 @@
 {
  "compilerOptions": {
    "target": "es5",
-    "lib": [
-      "dom",
-      "dom.iterable",
-      "esnext"
-    ],
+    "lib": ["dom", "dom.iterable", "esnext"],
    "allowJs": true,
    "skipLibCheck": true,
    "strict": true,
@@ -23,19 +19,10 @@
      }
    ],
    "paths": {
-      "@/*": [
-        "./*"
-      ]
+      "@/*": ["./*"]
    },
-    "forceConsistentCasingInFileNames": true,
+    "forceConsistentCasingInFileNames": true
  },
-  "include": [
-    "next-env.d.ts",
-    "**/*.ts",
-    "**/*.tsx",
-    ".next/types/**/*.ts"
-  ],
-  "exclude": [
-    "node_modules"
-  ]
-}
+  "include": ["next-env.d.ts", "**/*.ts", "**/*.tsx", ".next/types/**/*.ts"],
+  "exclude": ["node_modules"]
+}
@@ -7,8 +7,5 @@
    "esModuleInterop": true,
    "skipLibCheck": false
  },
-  "exclude": [
-    "templates",
-    "dist"
-  ]
-}
+  "exclude": ["templates", "dist"]
+}
@@ -28,7 +28,7 @@ module.exports = {
          "PINECONE_ENVIRONMENT",
          "PINECONE_PROJECT_ID",
          "PINECONE_INDEX_NAME",
-          "PINECONE_CHUNK_SIZE", 
+          "PINECONE_CHUNK_SIZE",
          "PINECONE_INDEX_NAME",

          "AZURE_OPENAI_API_KEY",
@@ -44,6 +44,8 @@ module.exports = {
          "NOTION_TOKEN",
          "MONGODB_URI",

+          "PG_CONNECTION_STRING",
+
          "https_proxy",
          "npm_config_user_agent",
          "NEXT_PUBLIC_CHAT_API",
@@ -19,5 +19,4 @@ sandbox:
    - pnpm eslint --fix {file_path}
    - pnpx ts-node --type-check {file_path}
    - pnpm test
-
 # Default Values: https://github.com/sweepai/sweep/blob/main/sweep.yaml
Author	SHA1	Message	Date
Marcus Schiesser	3d198d94a6	llamaindex@0.0.41	2024-01-08 15:18:24 +07:00
Thuc Pham	036c00db73	Feat: add postgres vectordb (#308 ) * feat: integrate create-llama with postgresql * fix: get data for verification before inserting * feat: show available vector DBs based on framework	2024-01-05 14:13:52 +08:00
Alex Yang	548f0687f1	feat(core): init support for Ollama (#305 )	2024-01-04 18:03:00 -06:00
Marcus Schiesser	f85c042a94	refactor: encapsulate node serialization	2024-01-04 10:19:18 +07:00
yisding	40e892f813	Oops adding back Alex's crypto changes	2024-01-04 10:19:18 +07:00
yisding	2381025306	move clone outside of toJSON I like structuredClone although we have an issue with AWS only supporting Node 16	2024-01-04 10:19:18 +07:00
Marcus Schiesser	a0909dc053	docs: updated how to create a mongodb vector index	2024-01-04 10:19:18 +07:00
Marcus Schiesser	853a14b7c7	fix: add test cases, ensure that a node's metadata is not modified and remove text if requested	2024-01-04 10:19:18 +07:00
yisding	b31e2d42eb	fixed recursive issue in metadataDictToNode	2024-01-04 10:19:18 +07:00
yisding	5b6ad9419f	fix mongo example	2024-01-04 10:19:18 +07:00
Marcus Schiesser	a60948a87e	fix: add format check to pre-commit	2024-01-03 10:39:54 +07:00
Alex Yang	8b420da753	style: prettier format (#304 )	2024-01-02 17:12:06 -06:00
Alex Yang	12c079b74a	refactor: remove unused deps (#303 )	2024-01-02 16:40:24 -06:00
Marcus Schiesser	c835f78dd0	docs: added changesets for core	2024-01-02 16:07:12 +07:00
Marcus Schiesser	d52eb9d4ee	fix[docs]: installation link (#302 )	2024-01-02 12:25:56 +08:00