Merge branch 'feat/integrate-postgresql-datasource' of github.com:run-llama/LlamaIndexTS into feat/integrate-postgresql-datasource

chore: resolve conflict
feat: use connection string for create llama
2026-07-13 22:17:48 -04:00 · 2024-01-04 15:37:12 +07:00 · 2024-01-04 15:36:58 +07:00 · 2024-01-04 15:36:21 +07:00 · 2024-01-04 15:36:21 +07:00 · 2024-01-04 15:36:20 +07:00
141 changed files with 50862 additions and 58974 deletions
@@ -0,0 +1,5 @@
+---
+"llamaindex": patch
+---
+
+Use compromise as sentence tokenizer
@@ -0,0 +1,5 @@
+---
+"llamaindex": patch
+---
+
+Removed pdf-parse, and directly use latest pdf.js
@@ -0,0 +1,5 @@
+---
+"llamaindex": patch
+---
+
+Added pinecone vector DB
@@ -36,6 +36,8 @@ jobs:
        working-directory: ./packages/create-llama
      - name: Run Playwright tests
        run: pnpm exec playwright test
+        env:
+          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        working-directory: ./packages/create-llama
      - uses: actions/upload-artifact@v3
        if: always()
@@ -17,9 +17,11 @@ jobs:
      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
-          node-version-file: '.nvmrc'
+          node-version-file: ".nvmrc"
          cache: "pnpm"
      - name: Install dependencies
        run: pnpm install
      - name: Run lint
        run: pnpm run lint
+      - name: Run Prettier
+        run: pnpm run format
@@ -12,7 +12,7 @@ jobs:
      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
-          node-version-file: '.nvmrc'
+          node-version-file: ".nvmrc"
          cache: "pnpm"
      - name: Install dependencies
        run: pnpm install
@@ -46,3 +46,6 @@ test-results/
 playwright-report/
 blob-report/
 playwright/.cache/
+
+# intellij
+**/.idea
@@ -1,5 +1,6 @@
 #!/usr/bin/env sh
 . "$(dirname -- "$0")/_/husky.sh"

+pnpm format
 pnpm lint
 npx lint-staged
@@ -0,0 +1,3 @@
+apps/docs/i18n
+pnpm-lock.yaml
+
@@ -5,5 +5,8 @@
  "[xml]": {
    "editor.defaultFormatter": "redhat.vscode-xml"
  },
-  "jest.rootPath": "./packages/core"
-}
+  "jest.rootPath": "./packages/core",
+  "[python]": {
+    "editor.defaultFormatter": "ms-python.black-formatter"
+  }
+}
@@ -8,7 +8,7 @@ Right now there are two packages of importance:

 packages/core which is the main NPM library llamaindex

-apps/simple is where the demo code lives
+examples is where the demo code lives

 ### Turborepo docs

@@ -47,7 +47,7 @@ We use Jest https://jestjs.io/ to write our test cases. Jest comes with a bunch

 ### Demo applications

-There is an existing ["simple"](/apps/simple/README.md) demos folder with mainly NodeJS scripts. Feel free to add additional demos to that folder. If you would like to try out your changes in the core package with a new demo, you need to run the build command in the README.
+There is an existing ["example"](/examples/README.md) demos folder with mainly NodeJS scripts. Feel free to add additional demos to that folder. If you would like to try out your changes in the core package with a new demo, you need to run the build command in the README.

 You can create new demo applications in the apps folder. Just run pnpm init in the folder after you create it to create its own package.json

@@ -56,7 +56,7 @@ You can create new demo applications in the apps folder. Just run pnpm init in t
 To install packages for a specific package or demo application, run

 ```
-pnpm add [NPM Package] --filter [package or application i.e. core or simple]
+pnpm add [NPM Package] --filter [package or application i.e. core or docs]
 ```

 To install packages for every package or application run
@@ -37,7 +37,7 @@ For more complex applications, our lower-level APIs allow advanced users to cust

 `npm install llamaindex`

-Our documentation includes [Installation Instructions](./installation.md) and a [Starter Tutorial](./starter.md) to build your first application.
+Our documentation includes [Installation Instructions](./installation.mdx) and a [Starter Tutorial](./starter.md) to build your first application.

 Once you're up and running, [High-Level Concepts](./concepts.md) has an overview of LlamaIndex's modular architecture. For more hands-on practical examples, look through our [End-to-End Tutorials](./end_to_end.md).

@@ -30,21 +30,21 @@ const config = {
  i18n: {
    defaultLocale: "en",
    locales: [
-    "en", 
-    "zh-Hans", 
-    "es",
-    "fr",
-    "de",
-    "ja",
-    "ko",
-    "pt",
-    "ar",
-    "it",
-    "tr",
-    "pl",
-    "nl",
-    "vi",
-    "th",
+      "en",
+      "zh-Hans",
+      "es",
+      "fr",
+      "de",
+      "ja",
+      "ko",
+      "pt",
+      "ar",
+      "it",
+      "tr",
+      "pl",
+      "nl",
+      "vi",
+      "th",
    ], // "fa", "ru", "ro", "sv", "hu", "cs", "el", "da", "fi", "he", "no", "hi", "in", "sl", "se", "sk", "uk", "bg", "hr", "lt", "lv", "et", "cat"
  },

@@ -66,10 +66,10 @@ const config = {
        },
        blog: false,
        gtag: {
-          trackingID: 'G-NB9B8LW9W5',
+          trackingID: "G-NB9B8LW9W5",
          anonymizeIP: true,
        },
-    }),
+      }),
    ],
  ],

@@ -39,7 +39,7 @@ LlamaIndex.TS هو إطار بيانات لتطبيقات LLM لاستيعاب

 `npm install llamaindex`

-تتضمن وثائقنا [تعليمات التثبيت](./installation.md) و[دليل البداية](./starter.md) لبناء تطبيقك الأول.
+تتضمن وثائقنا [تعليمات التثبيت](./installation.mdx) و[دليل البداية](./starter.md) لبناء تطبيقك الأول.

 بمجرد أن تكون جاهزًا وتعمل ، يحتوي [مفاهيم عالية المستوى](./concepts.md) على نظرة عامة على الهندسة المعمارية المتعددة المستويات لـ LlamaIndex. لمزيد من الأمثلة العملية التفصيلية ، يمكنك الاطلاع على [دروس النهاية إلى النهاية](./end_to_end.md).

@@ -41,7 +41,7 @@ LlamaIndex.TS предоставя основен набор от инструм

 `npm install llamaindex`

-Документацията ни включва [Инструкции за инсталиране](./installation.md) и [Урок за начинаещи](./starter.md), за да построите първото си приложение.
+Документацията ни включва [Инструкции за инсталиране](./installation.mdx) и [Урок за начинаещи](./starter.md), за да построите първото си приложение.

 След като сте готови, [Високо ниво концепции](./concepts.md) представя общ преглед на модулната архитектура на LlamaIndex. За повече практически примери, разгледайте нашите [Уроци от начало до край](./end_to_end.md).

@@ -39,7 +39,7 @@ Per a aplicacions més complexes, les nostres API de nivell inferior permeten al

 `npm install llamaindex`

-La nostra documentació inclou [Instruccions d'Instal·lació](./installation.md) i un [Tutorial d'Inici](./starter.md) per a construir la vostra primera aplicació.
+La nostra documentació inclou [Instruccions d'Instal·lació](./installation.mdx) i un [Tutorial d'Inici](./starter.md) per a construir la vostra primera aplicació.

 Un cop tingueu tot a punt, [Conceptes de Nivell Alt](./concepts.md) ofereix una visió general de l'arquitectura modular de LlamaIndex. Per a més exemples pràctics, consulteu els nostres [Tutorials de Principi a Fi](./end_to_end.md).

@@ -39,7 +39,7 @@ Pro složitější aplikace naše API na nižší úrovni umožňuje pokročilý

 `npm install llamaindex`

-Naše dokumentace obsahuje [Návod k instalaci](./installation.md) a [Úvodní tutoriál](./starter.md) pro vytvoření vaší první aplikace.
+Naše dokumentace obsahuje [Návod k instalaci](./installation.mdx) a [Úvodní tutoriál](./starter.md) pro vytvoření vaší první aplikace.

 Jakmile jste připraveni, [Vysokoúrovňové koncepty](./concepts.md) poskytují přehled o modulární architektuře LlamaIndexu. Pro více praktických příkladů se podívejte na naše [Tutoriály od začátku do konce](./end_to_end.md).

@@ -39,7 +39,7 @@ Til mere komplekse applikationer giver vores API'er på lavere niveau avancerede

 `npm install llamaindex`

-Vores dokumentation inkluderer [Installationsinstruktioner](./installation.md) og en [Starter Tutorial](./starter.md) til at bygge din første applikation.
+Vores dokumentation inkluderer [Installationsinstruktioner](./installation.mdx) og en [Starter Tutorial](./starter.md) til at bygge din første applikation.

 Når du er i gang, giver [Højniveaukoncepter](./concepts.md) et overblik over LlamaIndex's modulære arkitektur. For flere praktiske eksempler, kan du kigge igennem vores [End-to-End Tutorials](./end_to_end.md).

@@ -39,7 +39,7 @@ Für komplexere Anwendungen ermöglichen unsere APIs auf niedrigerer Ebene fortg

 `npm install llamaindex`

-Unsere Dokumentation enthält [Installationsanweisungen](./installation.md) und ein [Einführungstutorial](./starter.md), um Ihre erste Anwendung zu erstellen.
+Unsere Dokumentation enthält [Installationsanweisungen](./installation.mdx) und ein [Einführungstutorial](./starter.md), um Ihre erste Anwendung zu erstellen.

 Sobald Sie bereit sind, bietet [High-Level-Konzepte](./concepts.md) einen Überblick über die modulare Architektur von LlamaIndex. Für praktische Beispiele schauen Sie sich unsere [End-to-End-Tutorials](./end_to_end.md) an.

@@ -41,7 +41,7 @@ slug: /

 `npm install llamaindex`

-Η τεκμηρίωσή μας περιλαμβάνει [Οδηγίες Εγκατάστασης](./installation.md) και ένα [Εισαγωγικό Εκπαιδευτικό Πρόγραμμα](./starter.md) για να δημιουργήσετε την πρώτη σας εφαρμογή.
+Η τεκμηρίωσή μας περιλαμβάνει [Οδηγίες Εγκατάστασης](./installation.mdx) και ένα [Εισαγωγικό Εκπαιδευτικό Πρόγραμμα](./starter.md) για να δημιουργήσετε την πρώτη σας εφαρμογή.

 Αφού ξεκινήσετε, οι [Υψηλού Επιπέδου Έννοιες](./concepts.md) παρέχουν μια επισκόπηση της μοντουλαρισμένης αρχιτεκτονικής του LlamaIndex. Για περισσότερα πρακτικά παραδείγματα, ρίξτε μια ματιά στα [Ολοκληρωμένα Εκπαιδευτικά Προγράμματα](./end_to_end.md).

@@ -39,7 +39,7 @@ Para aplicaciones más complejas, nuestras API de nivel inferior permiten a los

 `npm install llamaindex`

-Nuestra documentación incluye [Instrucciones de instalación](./installation.md) y un [Tutorial de inicio](./starter.md) para construir tu primera aplicación.
+Nuestra documentación incluye [Instrucciones de instalación](./installation.mdx) y un [Tutorial de inicio](./starter.md) para construir tu primera aplicación.

 Una vez que estés en funcionamiento, [Conceptos de alto nivel](./concepts.md) ofrece una visión general de la arquitectura modular de LlamaIndex. Para obtener ejemplos prácticos más detallados, consulta nuestros [Tutoriales de extremo a extremo](./end_to_end.md).

@@ -39,7 +39,7 @@ Täpsemate rakenduste jaoks võimaldavad meie madalama taseme API-d edasijõudnu

 `npm install llamaindex`

-Meie dokumentatsioonis on [paigaldusjuhised](./installation.md) ja [algõpetus](./starter.md) oma esimese rakenduse loomiseks.
+Meie dokumentatsioonis on [paigaldusjuhised](./installation.mdx) ja [algõpetus](./starter.md) oma esimese rakenduse loomiseks.

 Kui olete valmis ja töötate, siis [kõrgtasemel kontseptsioonid](./concepts.md) annavad ülevaate LlamaIndexi moodularhitektuurist. Praktiliste näidete jaoks vaadake läbi meie [otsast lõpuni õpetused](./end_to_end.md).

@@ -41,7 +41,7 @@ API سطح بالای ما به کاربران مبتدی امکان استفا

 `npm install llamaindex`

-مستندات ما شامل [دستورالعمل نصب](./installation.md) و [آموزش شروع کار](./starter.md) برای ساخت اولین برنامه شما است.
+مستندات ما شامل [دستورالعمل نصب](./installation.mdx) و [آموزش شروع کار](./starter.md) برای ساخت اولین برنامه شما است.

 با راه اندازی و اجرا شدن، [مفاهیم سطح بالا](./concepts.md) یک نمای کلی از معماری ماژولار لاماایندکس را ارائه می دهد. برای مثال های عملی بیشتر، به [آموزش های پایان به پایان](./end_to_end.md) مراجعه کنید.

@@ -41,7 +41,7 @@ Monimutkaisempiin sovelluksiin tarjoamme matalamman tason API:t, jotka mahdollis

 `npm install llamaindex`

-Dokumentaatiostamme löydät [asennusohjeet](./installation.md) ja [aloitusopetusohjelman](./starter.md) ensimmäisen sovelluksesi rakentamiseen.
+Dokumentaatiostamme löydät [asennusohjeet](./installation.mdx) ja [aloitusopetusohjelman](./starter.md) ensimmäisen sovelluksesi rakentamiseen.

 Kun olet päässyt vauhtiin, [Korkean tason käsitteet](./concepts.md) antaa yleiskuvan LlamaIndexin modulaarisesta arkkitehtuurista. Lisää käytännön esimerkkejä löydät [Päästä päähän -opetusohjelmista](./end_to_end.md).

@@ -41,7 +41,7 @@ API הרמה הגבוהה שלנו מאפשר למשתמשים מתחילים ל

 `npm install llamaindex`

-התיעוד שלנו כולל [הוראות התקנה](./installation.md) ו[מדריך התחלה](./starter.md) לבניית היישום הראשון שלך.
+התיעוד שלנו כולל [הוראות התקנה](./installation.mdx) ו[מדריך התחלה](./starter.md) לבניית היישום הראשון שלך.

 כאשר אתה מוכן ורץ, [מושגים ברמה גבוהה](./concepts.md) מציג סקירה על ארכיטקטורה מודולרית של LlamaIndex. לדוגמאות פרקטיות יותר, עיין ב[מדריכים מתקדמים מתחילה ועד סוף](./end_to_end.md).

@@ -41,7 +41,7 @@ LlamaIndex.TS जावास्क्रिप्ट और TypeScript के

 `npm install llamaindex`

-हमारी दस्तावेज़ी में [स्थापना निर्देश](./installation.md) और [स्टार्टर ट्यूटोरियल](./starter.md) शामिल हैं, जिनका उपयोग करके आप अपना पहला एप्लिकेशन बना सकते हैं।
+हमारी दस्तावेज़ी में [स्थापना निर्देश](./installation.mdx) और [स्टार्टर ट्यूटोरियल](./starter.md) शामिल हैं, जिनका उपयोग करके आप अपना पहला एप्लिकेशन बना सकते हैं।

 एक बार जब आप शुरू हो जाएं, [उच्च स्तरीय अवधारणाएँ](./concepts.md) में LlamaIndex की मॉड्यूलर आर्किटेक्चर का अवलोकन है। अधिक हैंड्स-ऑन प्रैक्टिकल उदाहरणों के लिए, हमारे [एंड-टू-एंड ट्यूटोरियल](./end_to_end.md) को देखें।

@@ -39,7 +39,7 @@ Za složenije aplikacije, naše API-je niže razine omogućuju naprednim korisni

 `npm install llamaindex`

-Naša dokumentacija uključuje [Upute za instalaciju](./installation.md) i [Uvodni vodič](./starter.md) za izgradnju vaše prve aplikacije.
+Naša dokumentacija uključuje [Upute za instalaciju](./installation.mdx) i [Uvodni vodič](./starter.md) za izgradnju vaše prve aplikacije.

 Kada ste spremni za rad, [Visokorazinski koncepti](./concepts.md) pružaju pregled modularne arhitekture LlamaIndex-a. Za praktične primjere, pogledajte naše [Vodiče od početka do kraja](./end_to_end.md).

@@ -39,7 +39,7 @@ A komplexebb alkalmazásokhoz a mélyebb szintű API-k lehetővé teszik a halad

 `npm install llamaindex`

-A dokumentációnk tartalmazza a [Telepítési utasításokat](./installation.md) és egy [Kezdő útmutatót](./starter.md) az első alkalmazás létrehozásához.
+A dokumentációnk tartalmazza a [Telepítési utasításokat](./installation.mdx) és egy [Kezdő útmutatót](./starter.md) az első alkalmazás létrehozásához.

 Miután elindultál, a [Magas szintű fogalmak](./concepts.md) áttekintést ad a LlamaIndex moduláris architektúrájáról. További gyakorlati példákért tekintsd meg az [End-to-End útmutatóinkat](./end_to_end.md).

@@ -39,7 +39,7 @@ Untuk aplikasi yang lebih kompleks, API tingkat lebih rendah kami memungkinkan p

 `npm install llamaindex`

-Dokumentasi kami mencakup [Instruksi Instalasi](./installation.md) dan [Tutorial Awal](./starter.md) untuk membangun aplikasi pertama Anda.
+Dokumentasi kami mencakup [Instruksi Instalasi](./installation.mdx) dan [Tutorial Awal](./starter.md) untuk membangun aplikasi pertama Anda.

 Setelah Anda mulai, [Konsep Tingkat Tinggi](./concepts.md) memberikan gambaran tentang arsitektur modular LlamaIndex. Untuk contoh praktis yang lebih mendalam, lihat [Tutorial End-to-End](./end_to_end.md).

@@ -39,7 +39,7 @@ Per applicazioni più complesse, le nostre API di livello inferiore consentono a

 `npm install llamaindex`

-La nostra documentazione include le [Istruzioni di installazione](./installation.md) e un [Tutorial introduttivo](./starter.md) per creare la tua prima applicazione.
+La nostra documentazione include le [Istruzioni di installazione](./installation.mdx) e un [Tutorial introduttivo](./starter.md) per creare la tua prima applicazione.

 Una volta che sei pronto, i [Concetti di alto livello](./concepts.md) offrono una panoramica dell'architettura modulare di LlamaIndex. Per ulteriori esempi pratici, consulta i nostri [Tutorial end-to-end](./end_to_end.md).

@@ -39,7 +39,7 @@ LlamaIndex.TSは、JavaScriptとTypeScriptを使用してLLMアプリを構築

 `npm install llamaindex`

-私たちのドキュメントには、[インストール手順](./installation.md)と[スターターチュートリアル](./starter.md)が含まれており、最初のアプリケーションの構築をサポートします。
+私たちのドキュメントには、[インストール手順](./installation.mdx)と[スターターチュートリアル](./starter.md)が含まれており、最初のアプリケーションの構築をサポートします。

 一度準備ができたら、[ハイレベルなコンセプト](./concepts.md)では、LlamaIndexのモジュラーアーキテクチャの概要を説明しています。より実践的な例については、[エンドツーエンドのチュートリアル](./end_to_end.md)を参照してください。

@@ -39,7 +39,7 @@ LlamaIndex.TS는 JavaScript와 TypeScript로 LLM 앱을 개발하는 모든 사

 `npm install llamaindex`

-저희 문서에는 [설치 지침](./installation.md)과 [스타터 튜토리얼](./starter.md)이 포함되어 있어 첫 번째 애플리케이션을 빌드할 수 있습니다.
+저희 문서에는 [설치 지침](./installation.mdx)과 [스타터 튜토리얼](./starter.md)이 포함되어 있어 첫 번째 애플리케이션을 빌드할 수 있습니다.

 한 번 시작하면, [고수준 개념](./concepts.md)에서 LlamaIndex의 모듈식 아키텍처에 대한 개요를 확인할 수 있습니다. 더 많은 실전 예제를 원하신다면, [End-to-End 튜토리얼](./end_to_end.md)을 참조해주세요.

@@ -39,7 +39,7 @@ Sudėtingesnėms programoms mūsų žemesnio lygio API leidžia pažengusiems na

 `npm install llamaindex`

-Mūsų dokumentacija apima [įdiegimo instrukcijas](./installation.md) ir [pradžios vadovą](./starter.md), skirtą sukurti pirmąją aplikaciją.
+Mūsų dokumentacija apima [įdiegimo instrukcijas](./installation.mdx) ir [pradžios vadovą](./starter.md), skirtą sukurti pirmąją aplikaciją.

 Kai jau esate paleidę, [aukšto lygio konceptai](./concepts.md) pateikia apžvalgą apie LlamaIndex modularią architektūrą. Norėdami gauti daugiau praktinių pavyzdžių, peržiūrėkite mūsų [nuo pradžių iki pabaigos vadovus](./end_to_end.md).

@@ -41,7 +41,7 @@ Lielākām un sarežģītākām lietojumprogrammām mūsu zemāka līmeņa API

 `npm install llamaindex`

-Mūsu dokumentācijā ir iekļautas [Instalācijas instrukcijas](./installation.md) un [Sākuma pamācība](./starter.md), lai izveidotu savu pirmo lietojumprogrammu.
+Mūsu dokumentācijā ir iekļautas [Instalācijas instrukcijas](./installation.mdx) un [Sākuma pamācība](./starter.md), lai izveidotu savu pirmo lietojumprogrammu.

 Kad esat gatavs, [Augsta līmeņa koncepti](./concepts.md) sniedz pārskatu par LlamaIndex modulāro arhitektūru. Lai iegūtu vairāk praktisku piemēru, apskatiet mūsu [Galēji līdz galam pamācības](./end_to_end.md).

@@ -39,7 +39,7 @@ Voor complexere toepassingen stellen onze API's op lager niveau gevorderde gebru

 `npm install llamaindex`

-Onze documentatie bevat [Installatie-instructies](./installation.md) en een [Starterzelfstudie](./starter.md) om uw eerste toepassing te bouwen.
+Onze documentatie bevat [Installatie-instructies](./installation.mdx) en een [Starterzelfstudie](./starter.md) om uw eerste toepassing te bouwen.

 Zodra u aan de slag bent, geeft [Hoog-niveau Concepten](./concepts.md) een overzicht van de modulaire architectuur van LlamaIndex. Voor meer praktische voorbeelden kunt u onze [End-to-End Tutorials](./end_to_end.md) bekijken.

@@ -39,7 +39,7 @@ For mer komplekse applikasjoner lar våre lavnivå-APIer avanserte brukere tilpa

 `npm install llamaindex`

-Dokumentasjonen vår inkluderer [Installasjonsinstruksjoner](./installation.md) og en [Starterveiledning](./starter.md) for å bygge din første applikasjon.
+Dokumentasjonen vår inkluderer [Installasjonsinstruksjoner](./installation.mdx) og en [Starterveiledning](./starter.md) for å bygge din første applikasjon.

 Når du er oppe og kjører, gir [Høynivåkonsepter](./concepts.md) en oversikt over LlamaIndex sin modulære arkitektur. For mer praktiske eksempler, kan du se gjennom våre [End-to-End veiledninger](./end_to_end.md).

@@ -39,7 +39,7 @@ Dla bardziej zaawansowanych aplikacji nasze API na niższym poziomie umożliwia

 `npm install llamaindex`

-Nasza dokumentacja zawiera [Instrukcje instalacji](./installation.md) oraz [Samouczek dla początkujących](./starter.md), który pomoże Ci zbudować swoją pierwszą aplikację.
+Nasza dokumentacja zawiera [Instrukcje instalacji](./installation.mdx) oraz [Samouczek dla początkujących](./starter.md), który pomoże Ci zbudować swoją pierwszą aplikację.

 Gdy już będziesz gotowy, [Wysokopoziomowe koncepcje](./concepts.md) zawierają przegląd modułowej architektury LlamaIndex. Jeśli chcesz zobaczyć praktyczne przykłady, zapoznaj się z naszymi [Samouczkami od początku do końca](./end_to_end.md).

@@ -39,7 +39,7 @@ Para aplicativos mais complexos, nossas APIs de nível inferior permitem que usu

 `npm install llamaindex`

-Nossa documentação inclui [Instruções de Instalação](./installation.md) e um [Tutorial Inicial](./starter.md) para construir seu primeiro aplicativo.
+Nossa documentação inclui [Instruções de Instalação](./installation.mdx) e um [Tutorial Inicial](./starter.md) para construir seu primeiro aplicativo.

 Depois de estar pronto para começar, [Conceitos de Alto Nível](./concepts.md) oferece uma visão geral da arquitetura modular do LlamaIndex. Para exemplos práticos mais detalhados, consulte nossos [Tutoriais de Ponta a Ponta](./end_to_end.md).

@@ -39,7 +39,7 @@ Pentru aplicații mai complexe, API-urile noastre de nivel inferior permit utili

 `npm install llamaindex`

-Documentația noastră include [Instrucțiuni de instalare](./installation.md) și un [Tutorial de pornire](./starter.md) pentru a construi prima ta aplicație.
+Documentația noastră include [Instrucțiuni de instalare](./installation.mdx) și un [Tutorial de pornire](./starter.md) pentru a construi prima ta aplicație.

 Odată ce ai început, [Concepte de nivel înalt](./concepts.md) oferă o prezentare generală a arhitecturii modulare a LlamaIndex. Pentru mai multe exemple practice, consultă [Tutorialele de la cap la coadă](./end_to_end.md).

@@ -39,7 +39,7 @@ LlamaIndex.TS предоставляет основной набор инстр

 `npm install llamaindex`

-Наша документация включает [Инструкции по установке](./installation.md) и [Стартовое руководство](./starter.md) для создания вашего первого приложения.
+Наша документация включает [Инструкции по установке](./installation.mdx) и [Стартовое руководство](./starter.md) для создания вашего первого приложения.

 Когда вы начнете работу, [Высокоуровневые концепции](./concepts.md) предоставляют обзор модульной архитектуры LlamaIndex. Для более практических примеров руководство [Полный цикл руководств](./end_to_end.md) будет полезно.

@@ -39,7 +39,7 @@ Za složenije aplikacije, naše API-je na nižem nivou omogućavaju naprednim ko

 `npm install llamaindex`

-Naša dokumentacija uključuje [Uputstva za instalaciju](./installation.md) i [Uvodni tutorijal](./starter.md) za izgradnju vaše prve aplikacije.
+Naša dokumentacija uključuje [Uputstva za instalaciju](./installation.mdx) i [Uvodni tutorijal](./starter.md) za izgradnju vaše prve aplikacije.

 Kada ste spremni za rad, [Koncepti na visokom nivou](./concepts.md) pružaju pregled modularne arhitekture LlamaIndex-a. Za praktične primere, pogledajte naše [Vodiče od početka do kraja](./end_to_end.md).

@@ -39,7 +39,7 @@ Za bolj kompleksne aplikacije naša nizkonivojska API omogoča naprednim uporabn

 `npm install llamaindex`

-Naša dokumentacija vključuje [Navodila za namestitev](./installation.md) in [Vodič za začetek](./starter.md), ki vam pomagata zgraditi vašo prvo aplikacijo.
+Naša dokumentacija vključuje [Navodila za namestitev](./installation.mdx) in [Vodič za začetek](./starter.md), ki vam pomagata zgraditi vašo prvo aplikacijo.

 Ko ste pripravljeni, [Visokonivojski koncepti](./concepts.md) ponujajo pregled modularne arhitekture LlamaIndex-a. Za več praktičnih primerov si oglejte naše [Vodiče od začetka do konca](./end_to_end.md).

@@ -39,7 +39,7 @@ Pre zložitejšie aplikácie naša nižšia úroveň API umožňuje pokročilým

 `npm install llamaindex`

-Naša dokumentácia obsahuje [Inštalačné pokyny](./installation.md) a [Úvodný tutoriál](./starter.md) pre vytvorenie vašej prvej aplikácie.
+Naša dokumentácia obsahuje [Inštalačné pokyny](./installation.mdx) a [Úvodný tutoriál](./starter.md) pre vytvorenie vašej prvej aplikácie.

 Keď už máte všetko pripravené, [Vysokoúrovňové koncepty](./concepts.md) poskytujú prehľad o modulárnej architektúre LlamaIndexu. Pre viac praktických príkladov si prečítajte naše [Tutoriály od začiatku do konca](./end_to_end.md).

@@ -39,7 +39,7 @@ För mer komplexa applikationer tillåter våra lägre nivå-API:er avancerade a

 `npm install llamaindex`

-Vår dokumentation inkluderar [Installationsinstruktioner](./installation.md) och en [Starterhandledning](./starter.md) för att bygga din första applikation.
+Vår dokumentation inkluderar [Installationsinstruktioner](./installation.mdx) och en [Starterhandledning](./starter.md) för att bygga din första applikation.

 När du är igång, ger [Högnivåkoncept](./concepts.md) en översikt över LlamaIndex modulära arkitektur. För mer praktiska exempel, titta igenom våra [Steg-för-steg handledningar](./end_to_end.md).

@@ -39,7 +39,7 @@ API ระดับสูงของเราช่วยให้ผู้ใ

 `npm install llamaindex`

-เอกสารของเราประกอบด้วย[คำแนะนำการติดตั้ง](./installation.md)และ[บทแนะนำเบื้องต้น](./starter.md)เพื่อสร้างแอปพลิเคชันครั้งแรกของคุณ
+เอกสารของเราประกอบด้วย[คำแนะนำการติดตั้ง](./installation.mdx)และ[บทแนะนำเบื้องต้น](./starter.md)เพื่อสร้างแอปพลิเคชันครั้งแรกของคุณ

 เมื่อคุณเริ่มใช้งานแล้ว [แนวคิดระดับสูง](./concepts.md) มีภาพรวมของสถาปัตยกรรมแบบโมดูลของ LlamaIndex สำหรับตัวอย่างที่เป็นปฏิบัติจริงมากขึ้น โปรดดูที่ [บทแนะนำจบสู่จบ](./end_to_end.md) เพื่อตัวอย่างที่ใช้งานได้จริง

@@ -41,7 +41,7 @@ Daha karmaşık uygulamalar için, düşük seviyeli API'larımız, gelişmiş k

 `npm install llamaindex`

-Dökümantasyonumuz, [Kurulum Talimatları](./installation.md) ve ilk uygulamanızı oluşturmanız için bir [Başlangıç Kılavuzu](./starter.md) içerir.
+Dökümantasyonumuz, [Kurulum Talimatları](./installation.mdx) ve ilk uygulamanızı oluşturmanız için bir [Başlangıç Kılavuzu](./starter.md) içerir.

 Çalışmaya başladıktan sonra, [Yüksek Düzeyli Kavramlar](./concepts.md) LlamaIndex'in modüler mimarisinin bir genel bakışını sunar. Daha fazla pratik örnek için [Uçtan Uca Öğreticilerimize](./end_to_end.md) göz atabilirsiniz.

@@ -39,7 +39,7 @@ LlamaIndex.TS надає основний набір інструментів,

 `npm install llamaindex`

-Наша документація містить [Інструкції з встановлення](./installation.md) та [Посібник для початківців](./starter.md) для створення вашої першої програми.
+Наша документація містить [Інструкції з встановлення](./installation.mdx) та [Посібник для початківців](./starter.md) для створення вашої першої програми.

 Після того, як ви розпочнете роботу, [Високорівневі концепції](./concepts.md) містить огляд модульної архітектури LlamaIndex. Для більш практичних прикладів роботи, перегляньте наші [Посібники з кінця в кінець](./end_to_end.md).

@@ -39,7 +39,7 @@ API cấp cao của chúng tôi cho phép người dùng mới bắt đầu sử

 `npm install llamaindex`

-Tài liệu của chúng tôi bao gồm [Hướng dẫn cài đặt](./installation.md) và [Hướng dẫn bắt đầu](./starter.md) để xây dựng ứng dụng đầu tiên của bạn.
+Tài liệu của chúng tôi bao gồm [Hướng dẫn cài đặt](./installation.mdx) và [Hướng dẫn bắt đầu](./starter.md) để xây dựng ứng dụng đầu tiên của bạn.

 Khi bạn đã sẵn sàng, [Khái niệm cấp cao](./concepts.md) cung cấp một cái nhìn tổng quan về kiến trúc mô-đun của LlamaIndex. Để có thêm ví dụ thực tế, hãy xem qua [Hướng dẫn từ đầu đến cuối](./end_to_end.md).

@@ -37,7 +37,7 @@ LlamaIndex.TS 提供了一套核心工具，对于任何使用JavaScript和TypeS

 `npm install llamaindex`

-我们的文档包括[安装说明](./installation.md)和一个[入门教程](./starter.md)，帮助你构建第一个应用程序。
+我们的文档包括[安装说明](./installation.mdx)和一个[入门教程](./starter.md)，帮助你构建第一个应用程序。

 一旦你开始运行，[高级概念](./concepts.md)有一个LlamaIndex模块化架构的概览。更多实践例子，请浏览我们的[端到端教程](./end_to_end.md)。

@@ -39,7 +39,7 @@ LlamaIndex.TS 提供了一組核心工具，對於使用 JavaScript 和 TypeScri

 `npm install llamaindex`

-我們的文檔包括[安裝說明](./installation.md)和[入門教程](./starter.md)，以構建您的第一個應用程序。
+我們的文檔包括[安裝說明](./installation.mdx)和[入門教程](./starter.md)，以構建您的第一個應用程序。

 一旦您開始運行，[高級概念](./concepts.md)提供了 LlamaIndex 模塊化架構的概述。如果需要更多實際的操作示例，請查看我們的[端到端教程](./end_to_end.md)。

@@ -1,15 +1,16 @@
 import { program } from "commander";
-import { AudioTranscriptReader, TranscribeParams, VectorStoreIndex } from "llamaindex";
+import {
+  AudioTranscriptReader,
+  TranscribeParams,
+  VectorStoreIndex,
+} from "llamaindex";
 import { stdin as input, stdout as output } from "node:process";
 // readline/promises is still experimental so not in @types/node yet
 // @ts-ignore
 import readline from "node:readline/promises";

 program
-  .option(
-    "-a, --audio [string]",
-    "URL or path of the audio file to transcribe",
-  )
+  .option("-a, --audio [string]", "URL or path of the audio file to transcribe")
  .option("-i, --transcript-id [string]", "ID of the AssemblyAI transcript")
  .action(async (options) => {
    if (!process.env.ASSEMBLYAI_API_KEY) {
@@ -26,9 +27,7 @@ program
    } else if (options.transcriptId) {
      params = options.transcriptId;
    } else {
-      console.log(
-        "You must provide either an --audio or a --transcript-id",
-      );
+      console.log("You must provide either an --audio or a --transcript-id");
      return;
    }

@@ -1,29 +1,25 @@
+import { Document, SimpleMongoReader, VectorStoreIndex } from "llamaindex";
 import { MongoClient } from "mongodb";
-import { Document } from "../../packages/core/src/Node";
-import { VectorStoreIndex } from "../../packages/core/src/indices";
-import { SimpleMongoReader } from "../../packages/core/src/readers/SimpleMongoReader";

 import { stdin as input, stdout as output } from "node:process";
 import readline from "node:readline/promises";

 async function main() {
  //Dummy test code
-  const query: object = { _id: "waldo" };
-  const options: object = {};
-  const projections: object = { embedding: 0 };
+  const filterQuery = {};
  const limit: number = Infinity;
-  const uri: string = process.env.MONGODB_URI ?? "fake_uri";
+  const uri: string = process.env.MONGODB_URI ?? "mongodb://localhost:27017";
  const client: MongoClient = new MongoClient(uri);

  //Where the real code starts
  const MR = new SimpleMongoReader(client);
  const documents: Document[] = await MR.loadData(
-    "data",
-    "posts",
-    1,
-    {},
-    options,
-    projections,
+    "db",
+    "collection",
+    ["text"],
+    "",
+    filterQuery,
+    limit,
  );

  //
@@ -43,7 +39,7 @@ async function main() {
  // console.log(nodes);

  //
-  //Making Vector Store from documents
+  // Making Vector Store from documents
  //

  const index = await VectorStoreIndex.fromDocuments(documents);
@@ -45,6 +45,39 @@ async function loadAndIndex() {
  await client.close();
 }

-loadAndIndex();
+/**
+ * This method is document in https://www.mongodb.com/docs/atlas/atlas-search/create-index/#create-an-fts-index-programmatically
+ * But, while testing a 'CommandNotFound' error occurred, so we're not using this here.
+ */
+async function createSearchIndex() {
+  const client = new MongoClient(mongoUri);
+  const database = client.db(databaseName);
+  const collection = database.collection(vectorCollectionName);
+
+  // define your Atlas Search index
+  const index = {
+    name: indexName,
+    definition: {
+      /* search index definition fields */
+      mappings: {
+        dynamic: true,
+        fields: [
+          {
+            type: "vector",
+            path: "embedding",
+            numDimensions: 1536,
+            similarity: "cosine",
+          },
+        ],
+      },
+    },
+  };
+  // run the helper method
+  const result = await collection.createSearchIndex(index);
+  console.log("Successfully created search index:", result);
+  await client.close();
+}
+
+loadAndIndex().catch(console.error);

 // you can't query your index yet because you need to create a vector search index in mongodb's UI now
@@ -76,7 +76,7 @@ Now if all has gone well you should be able to log in to the Mongo Atlas UI and

 Now it's time to create the vector search index so that you can query the data.
 It's not yet possible to programmatically create a vector search index using the [`createIndex`](https://www.mongodb.com/docs/manual/reference/method/db.collection.createIndex/) function, therefore we have to create one manually in the UI.
-To do so, first, click the Search tab, and then click "Create Search Index":
+To do so, first, click the 'Atlas Search' tab, and then click "Create Search Index":

 ![MongoDB Atlas create search index](./docs/4_search_tab.png)

@@ -88,16 +88,14 @@ Now under "database and collection" select `tiny_tweets_db` and within that sele

 ```json
 {
-  "mappings": {
-    "dynamic": true,
-    "fields": {
-      "embedding": {
-        "dimensions": 1536,
-        "similarity": "cosine",
-        "type": "knnVector"
-      }
+  "fields": [
+    {
+      "type": "vector",
+      "path": "embedding",
+      "numDimensions": 1536,
+      "similarity": "cosine"
    }
-  }
+  ]
 }
 ```

@@ -14,6 +14,19 @@ You'll also need a value for OPENAI_API_KEY in your environment.

 **NOTE:** Using `--rm` in the example docker command above means that the vector store will be deleted every time the container is stopped. For production purposes, use a volume to ensure persistence across restarts.

+## Use a database on cloud
+
+We recommend using a managed database service on cloud. For example, you can use [Timescale](https://docs.timescale.com/use-timescale/latest/services/create-a-service/?ref=timescale.com) to create a PostgreSQL database instance. You can then run the following command to set up environment variables for the database connection:
+
+```bash
+export PGHOST=<your database host>
+export PGUSER=<your database user>
+export PGPASSWORD=<your database password>
+export PGDATABASE=<your database name>
+export PGPORT=<your database port>
+export OPENAI_API_KEY=<your openai api key>s
+```
+
 ## Setup and Loading Docs

 Read and follow the instructions in the README.md file located one directory up to make sure your JS/TS dependencies are set up. The commands listed below are also run from that parent directory.
@@ -0,0 +1,33 @@
+# Pinecone Vector Store
+
+There are two scripts available here: load-docs.ts and query.ts
+
+## Prerequisites
+
+You'll need a Pinecone account, project, and index. Pinecone does not allow automatic creation of indexes on the free plan,
+so this vector store does not check and create the index (unlike, e.g., the PGVectorStore)
+
+Set the **PINECONE_API_KEY** and **PINECONE_ENVIRONMENT** environment variables to match your specific values. You will likely also need to set **PINECONE_INDEX_NAME**, unless your
+index is the default value "llama".
+
+You'll also need a value for OPENAI_API_KEY in your environment.
+
+## Setup and Loading Docs
+
+Read and follow the instructions in the README.md file located one directory up to make sure your JS/TS dependencies are set up. The commands listed below are also run from that parent directory.
+
+To import documents and save the embedding vectors to your database:
+
+> `npx ts-node pinecone-vector-store/load-docs.ts data`
+
+where data is the directory containing your input files. Using the _data_ directory in the example above will read all of the files in that directory using the llamaindexTS default readers for each file type.
+
+**NOTE**: Sending text chunks as part of the Pinecone metadata means that upsert API calls can get arbitrarily large. Set the **PINECONE_CHUNK_SIZE** environment variable to a smaller value if the load script fails
+
+## RAG Querying
+
+To query using the resulting vector store:
+
+> `npx ts-node pinecone-vector-store/query.ts`
+
+The script will prompt for a question, then process and present the answer using the PineconeVectorStore data and your OpenAI API key. It will continue to prompt until you enter `q`, `quit` or `exit` as the next query.
@@ -0,0 +1,66 @@
+// load-docs.ts
+import fs from "fs/promises";
+import {
+  PineconeVectorStore,
+  SimpleDirectoryReader,
+  storageContextFromDefaults,
+  VectorStoreIndex,
+} from "llamaindex";
+
+async function getSourceFilenames(sourceDir: string) {
+  return await fs
+    .readdir(sourceDir)
+    .then((fileNames) => fileNames.map((file) => sourceDir + "/" + file));
+}
+
+function callback(
+  category: string,
+  name: string,
+  status: any,
+  message: string = "",
+): boolean {
+  console.log(category, name, status, message);
+  return true;
+}
+
+async function main(args: any) {
+  const sourceDir: string = args.length > 2 ? args[2] : "../data";
+
+  console.log(`Finding documents in ${sourceDir}`);
+  const fileList = await getSourceFilenames(sourceDir);
+  const count = fileList.length;
+  console.log(`Found ${count} files`);
+
+  console.log(`Importing contents from ${count} files in ${sourceDir}`);
+  var fileName = "";
+  try {
+    // Passing callback fn to the ctor here
+    // will enable looging to console.
+    // See callback fn, defined above.
+    const rdr = new SimpleDirectoryReader(callback);
+    const docs = await rdr.loadData({ directoryPath: sourceDir });
+
+    const pcvs = new PineconeVectorStore();
+
+    const ctx = await storageContextFromDefaults({ vectorStore: pcvs });
+
+    console.debug("  - creating vector store");
+    const index = await VectorStoreIndex.fromDocuments(docs, {
+      storageContext: ctx,
+    });
+    console.debug("  - done.");
+  } catch (err) {
+    console.error(fileName, err);
+    console.log(
+      "If your PineconeVectorStore connection failed, make sure to set env vars for PINECONE_API_KEY and PINECONE_ENVIRONMENT.  If the upserts failed, try setting PINECONE_CHUNK_SIZE to limit the content sent per chunk",
+    );
+    process.exit(1);
+  }
+
+  console.log(
+    "Done. Try running query.ts to ask questions against the imported embeddings.",
+  );
+  process.exit(0);
+}
+
+main(process.argv).catch((err) => console.error(err));
@@ -0,0 +1,67 @@
+import {
+  PineconeVectorStore,
+  VectorStoreIndex,
+  serviceContextFromDefaults,
+} from "llamaindex";
+
+async function main() {
+  const readline = require("readline").createInterface({
+    input: process.stdin,
+    output: process.stdout,
+  });
+
+  try {
+    const pcvs = new PineconeVectorStore();
+
+    const ctx = serviceContextFromDefaults();
+    const index = await VectorStoreIndex.fromVectorStore(pcvs, ctx);
+
+    // Query the index
+    const queryEngine = await index.asQueryEngine();
+
+    let question = "";
+    while (!isQuit(question)) {
+      question = await getUserInput(readline);
+
+      if (isQuit(question)) {
+        readline.close();
+        process.exit(0);
+      }
+
+      try {
+        const answer = await queryEngine.query(question);
+        console.log(answer.response);
+      } catch (error) {
+        console.error("Error:", error);
+      }
+    }
+  } catch (err) {
+    console.error(err);
+    console.log(
+      "If your PineconeVectorStore connection failed, make sure to set env vars for PINECONE_API_KEY and PINECONE_ENVIRONMENT.",
+    );
+    process.exit(1);
+  }
+}
+
+function isQuit(question: string) {
+  return ["q", "quit", "exit"].includes(question.trim().toLowerCase());
+}
+
+// Function to get user input as a promise
+function getUserInput(readline: any): Promise<string> {
+  return new Promise((resolve) => {
+    readline.question(
+      "What would you like to know?\n>",
+      (userInput: string) => {
+        resolve(userInput);
+      },
+    );
+  });
+}
+
+main()
+  .catch(console.error)
+  .finally(() => {
+    process.exit(1);
+  });
@@ -3,7 +3,8 @@
  "scripts": {
    "build": "turbo run build",
    "dev": "turbo run dev",
-    "format": "prettier --write \"**/*.{js,jsx,ts,tsx,md}\"",
+    "format": "prettier --ignore-unknown --cache --check .",
+    "format:write": "prettier --ignore-unknown --write .",
    "lint": "turbo run lint",
    "prepare": "husky install",
    "test": "turbo run test",
@@ -11,18 +12,18 @@
    "publish-snapshot": "turbo run build lint test --filter=\"!docs\" && changeset version --snapshot && changeset publish"
  },
  "devDependencies": {
-    "@changesets/cli": "^2.26.2",
-    "@turbo/gen": "^1.10.16",
-    "@types/jest": "^29.5.10",
-    "eslint": "^8.54.0",
+    "@changesets/cli": "^2.27.1",
+    "@turbo/gen": "^1.11.2",
+    "@types/jest": "^29.5.11",
+    "eslint": "^8.56.0",
    "eslint-config-custom": "workspace:*",
    "husky": "^8.0.3",
    "jest": "^29.7.0",
-    "lint-staged": "^15.1.0",
-    "prettier": "^3.1.0",
+    "lint-staged": "^15.2.0",
+    "prettier": "^3.1.1",
    "prettier-plugin-organize-imports": "^3.2.4",
    "ts-jest": "^29.1.1",
-    "turbo": "^1.10.16"
+    "turbo": "^1.11.2"
  },
  "packageManager": "pnpm@8.10.5+sha256.a4bd9bb7b48214bbfcd95f264bd75bb70d100e5d4b58808f5cd6ab40c6ac21c5",
  "pnpm": {
@@ -7,9 +7,10 @@
    "@datastax/astra-db-ts": "^0.1.2",
    "@mistralai/mistralai": "^0.0.7",
    "@notionhq/client": "^2.2.14",
+    "@pinecone-database/pinecone": "^1.1.2",
    "@xenova/transformers": "^2.10.0",
    "assemblyai": "^4.0.0",
-    "crypto-js": "^4.2.0",
+    "compromise": "^14.10.1",
    "file-type": "^18.7.0",
    "js-tiktoken": "^1.0.8",
    "lodash": "^4.17.21",
@@ -19,7 +20,7 @@
    "notion-md-crawler": "^0.0.2",
    "openai": "^4.20.1",
    "papaparse": "^5.4.1",
-    "pdf-parse": "^1.1.1",
+    "pdfjs-dist": "4.0.269",
    "pg": "^8.11.3",
    "pgvector": "^0.1.5",
    "portkey-ai": "^0.1.16",
@@ -30,12 +31,10 @@
    "wink-nlp": "^1.14.3"
  },
  "devDependencies": {
-    "@types/crypto-js": "^4.2.1",
    "@types/jest": "^29.5.11",
    "@types/lodash": "^4.14.202",
    "@types/node": "^18.19.2",
    "@types/papaparse": "^5.3.14",
-    "@types/pdf-parse": "^1.1.4",
    "@types/pg": "^8.10.9",
    "@types/uuid": "^9.0.7",
    "node-stdlib-browser": "^1.2.0",
@@ -1,4 +1,5 @@
-import CryptoJS from "crypto-js";
+import _ from "lodash";
+import { createHash } from "node:crypto";
 import path from "path";
 import { v4 as uuidv4 } from "uuid";

@@ -141,12 +142,26 @@ export abstract class BaseNode<T extends Metadata = Metadata> {
  }

  /**
-   * Used with built in JSON.stringify
-   * @returns
+   * Called by built in JSON.stringify (see https://javascript.info/json)
+   * Properties are read-only as they are not deep-cloned (not necessary for stringification).
+   * @see toMutableJSON - use to return a mutable JSON instead
   */
  toJSON(): Record<string, any> {
    return { ...this, type: this.getType() };
  }
+
+  clone(): BaseNode {
+    return jsonToNode(this.toMutableJSON()) as BaseNode;
+  }
+
+  /**
+   * Converts the object to a JSON representation.
+   * Properties can be safely modified as a deep clone of the properties are created.
+   * @return {Record<string, any>} - The JSON representation of the object.
+   */
+  toMutableJSON(): Record<string, any> {
+    return _.cloneDeep(this.toJSON());
+  }
 }

 /**
@@ -177,13 +192,13 @@ export class TextNode<T extends Metadata = Metadata> extends BaseNode<T> {
   * @returns
   */
  generateHash() {
-    const hashFunction = CryptoJS.algo.SHA256.create();
+    const hashFunction = createHash("sha256");
    hashFunction.update(`type=${this.getType()}`);
    hashFunction.update(
      `startCharIdx=${this.startCharIdx} endCharIdx=${this.endCharIdx}`,
    );
    hashFunction.update(this.getContent(MetadataMode.ALL));
-    return hashFunction.finalize().toString(CryptoJS.enc.Base64);
+    return hashFunction.digest("base64");
  }

  getType(): ObjectType {
@@ -1,5 +1,6 @@
+import nlp from "compromise";
+import { EOL } from "node:os";
 // GitHub translated
-
 import { globalsHelper } from "./GlobalsHelper";
 import { DEFAULT_CHUNK_OVERLAP, DEFAULT_CHUNK_SIZE } from "./constants";

@@ -18,34 +19,38 @@ class TextSplit {

 type SplitRep = { text: string; numTokens: number };

-/**
- * Tokenizes sentences. Suitable for English and most European languages.
- * @param text
- * @returns
- */
-export const englishSentenceTokenizer = (text: string) => {
-  // The first part is a lazy match for any character.
-  return text.match(/.+?[.?!]+[\])'"`’”]*(?:\s|$)|.+/g);
+export const defaultSentenceTokenizer = (text: string): string[] => {
+  return nlp(text)
+    .sentences()
+    .json()
+    .map((sentence: any) => sentence.text);
 };

+// Refs: https://github.com/fxsjy/jieba/issues/575#issuecomment-359637511
+const resentencesp =
+  /([﹒﹔﹖﹗．；。！？]["’”」』]{0,2}|：(?=["‘“「『]{1,2}|$))/;
 /**
- * Tokenizes sentences. Suitable for Chinese, Japanese, and Korean.
+ * Tokenizes sentences. Suitable for Chinese, Japanese, and Korean. Use instead of `defaultSentenceTokenizer`.
 * @param text
- * @returns
+ * @returns string[]
 */
-export const cjkSentenceTokenizer = (text: string) => {
-  // Accepts english style sentence endings with space and
-  // CJK style sentence endings with no space.
-  return text.match(
-    /.+?[.?!]+[\])'"`’”]*(?:\s|$)|.+?[。？！]+[\])'"`’”]*(?:\s|$)?|.+/g,
-  );
-};
+export function cjkSentenceTokenizer(sentence: string): string[] {
+  const slist = [];
+  const parts = sentence.split(resentencesp);

-export const unixLineSeparator = "\n";
-export const windowsLineSeparator = "\r\n";
-export const unixParagraphSeparator = unixLineSeparator + unixLineSeparator;
-export const windowsParagraphSeparator =
-  windowsLineSeparator + windowsLineSeparator;
+  for (let i = 0; i < parts.length; i++) {
+    const part = parts[i];
+    if (resentencesp.test(part) && slist.length > 0) {
+      slist[slist.length - 1] += part;
+    } else if (part) {
+      slist.push(part);
+    }
+  }
+
+  return slist.filter((s) => s.length > 0);
+}
+
+export const defaultParagraphSeparator = EOL + EOL + EOL;

 // In theory there's also Mac style \r only, but it's pre-OSX and I don't think
 // many documents will use it.
@@ -61,7 +66,7 @@ export class SentenceSplitter {
  private tokenizer: any;
  private tokenizerDecoder: any;
  private paragraphSeparator: string;
-  private chunkingTokenizerFn: (text: string) => RegExpMatchArray | null;
+  private chunkingTokenizerFn: (text: string) => string[];
  private splitLongSentences: boolean;

  constructor(options?: {
@@ -70,7 +75,7 @@ export class SentenceSplitter {
    tokenizer?: any;
    tokenizerDecoder?: any;
    paragraphSeparator?: string;
-    chunkingTokenizerFn?: (text: string) => RegExpMatchArray | null;
+    chunkingTokenizerFn?: (text: string) => string[];
    splitLongSentences?: boolean;
  }) {
    const {
@@ -78,8 +83,8 @@ export class SentenceSplitter {
      chunkOverlap = DEFAULT_CHUNK_OVERLAP,
      tokenizer = null,
      tokenizerDecoder = null,
-      paragraphSeparator = unixParagraphSeparator,
-      chunkingTokenizerFn = undefined,
+      paragraphSeparator = defaultParagraphSeparator,
+      chunkingTokenizerFn,
      splitLongSentences = false,
    } = options ?? {};

@@ -97,7 +102,7 @@ export class SentenceSplitter {
      tokenizerDecoder ?? globalsHelper.tokenizerDecoder();

    this.paragraphSeparator = paragraphSeparator;
-    this.chunkingTokenizerFn = chunkingTokenizerFn ?? englishSentenceTokenizer;
+    this.chunkingTokenizerFn = chunkingTokenizerFn ?? defaultSentenceTokenizer;
    this.splitLongSentences = splitLongSentences;
  }

@@ -222,15 +227,17 @@ export class SentenceSplitter {
        curChunkTokens + newSentenceSplits[i].numTokens >
        effectiveChunkSize
      ) {
-        // push curent doc list to docs
-        docs.push(
-          new TextSplit(
-            curChunkSentences
-              .map((sentence) => sentence.text)
-              .join(" ")
-              .trim(),
-          ),
-        );
+        if (curChunkSentences.length > 0) {
+          // push curent doc list to docs
+          docs.push(
+            new TextSplit(
+              curChunkSentences
+                .map((sentence) => sentence.text)
+                .join(" ")
+                .trim(),
+            ),
+          );
+        }

        const lastChunkSentences = curChunkSentences;

@@ -9,7 +9,6 @@ export * from "./PromptHelper";
 export * from "./QueryEngine";
 export * from "./QuestionGenerator";
 export * from "./Response";
-export * from "./synthesizers";
 export * from "./Retriever";
 export * from "./ServiceContext";
 export * from "./TextSplitter";
@@ -29,3 +28,4 @@ export * from "./readers/SimpleDirectoryReader";
 export * from "./readers/SimpleMongoReader";
 export * from "./readers/base";
 export * from "./storage";
+export * from "./synthesizers";
@@ -4,7 +4,6 @@ import {
  ImageNode,
  MetadataMode,
  ObjectType,
-  jsonToNode,
  splitNodesByType,
 } from "../../Node";
 import { BaseQueryEngine, RetrieverQueryEngine } from "../../QueryEngine";
@@ -24,7 +23,7 @@ import {
 } from "../../storage/StorageContext";
 import { BaseIndexStore } from "../../storage/indexStore/types";
 import { VectorStore } from "../../storage/vectorStore/types";
-import { ResponseSynthesizer, BaseSynthesizer } from "../../synthesizers";
+import { BaseSynthesizer } from "../../synthesizers";
 import {
  BaseIndex,
  BaseIndexInit,
@@ -278,7 +277,7 @@ export class VectorStoreIndex extends BaseIndex<IndexDict> {
        type === ObjectType.INDEX ||
        type === ObjectType.IMAGE
      ) {
-        const nodeWithoutEmbedding = jsonToNode(nodes[i].toJSON());
+        const nodeWithoutEmbedding = nodes[i].clone();
        nodeWithoutEmbedding.embedding = undefined;
        this.indexStruct.addNode(nodeWithoutEmbedding, newIds[i]);
        this.docStore.addDocuments([nodeWithoutEmbedding], true);
@@ -10,6 +10,7 @@ import {

 import { ChatCompletionMessageParam } from "openai/resources";
 import { LLMOptions } from "portkey-ai";
+import { MessageContent } from "../ChatEngine";
 import { globalsHelper, Tokenizers } from "../GlobalsHelper";
 import {
  ANTHROPIC_AI_PROMPT,
@@ -27,7 +28,6 @@ import {
 import { getOpenAISession, OpenAISession } from "./openai";
 import { getPortkeySession, PortkeySession } from "./portkey";
 import { ReplicateSession } from "./replicate";
-import { MessageContent } from "../ChatEngine";

 export type MessageType =
  | "user"
@@ -1,7 +1,7 @@
 import mammoth from "mammoth";
 import { Document } from "../Node";
-import { DEFAULT_FS } from "../storage/constants";
 import { GenericFileSystem } from "../storage/FileSystem";
+import { DEFAULT_FS } from "../storage/constants";
 import { BaseReader } from "./base";

 export class DocxReader implements BaseReader {
@@ -1,4 +1,3 @@
-import pdfParse from "pdf-parse";
 import { Document } from "../Node";
 import { GenericFileSystem } from "../storage/FileSystem";
 import { DEFAULT_FS } from "../storage/constants";
@@ -12,8 +11,78 @@ export class PDFReader implements BaseReader {
    file: string,
    fs: GenericFileSystem = DEFAULT_FS,
  ): Promise<Document[]> {
-    const dataBuffer = (await fs.readFile(file)) as any;
-    const data = await pdfParse(dataBuffer);
-    return [new Document({ text: data.text, id_: file })];
+    const content = (await fs.readFile(file)) as any;
+    if (!(content instanceof Buffer)) {
+      console.warn(`PDF File ${file} can only be loaded using the Node FS`);
+      return [];
+    }
+    const data = new Uint8Array(
+      content.buffer,
+      content.byteOffset,
+      content.byteLength,
+    );
+    const pdf = await readPDF(data);
+    return [new Document({ text: pdf.text, id_: file })];
  }
 }
+
+// NOTE: the following code is taken from https://www.npmjs.com/package/pdf-parse and modified
+async function readPage(pageData: any) {
+  //check documents https://mozilla.github.io/pdf.js/
+  const textContent = await pageData.getTextContent({
+    includeMarkedContent: false,
+  });
+
+  let lastY = null,
+    text = "";
+  //https://github.com/mozilla/pdf.js/issues/8963
+  //https://github.com/mozilla/pdf.js/issues/2140
+  //https://gist.github.com/hubgit/600ec0c224481e910d2a0f883a7b98e3
+  //https://gist.github.com/hubgit/600ec0c224481e910d2a0f883a7b98e3
+  for (const item of textContent.items) {
+    if (lastY == item.transform[5] || !lastY) {
+      text += item.str;
+    } else {
+      text += "\n" + item.str;
+    }
+    lastY = item.transform[5];
+  }
+  return text;
+}
+
+const PDF_DEFAULT_OPTIONS = {
+  max: 0,
+};
+
+async function readPDF(data: Uint8Array, options = PDF_DEFAULT_OPTIONS) {
+  const { getDocument, version } = await import("pdfjs-dist");
+
+  const doc = await getDocument({ data }).promise;
+  const metaData = await doc.getMetadata().catch(() => null);
+  const counter =
+    options.max === 0 ? doc.numPages : Math.max(options.max, doc.numPages);
+
+  let text = "";
+
+  for (let i = 1; i <= counter; i++) {
+    try {
+      const pageData = await doc.getPage(i);
+      const pageText = await readPage(pageData);
+
+      text += `\n\n${pageText}`;
+    } catch (err) {
+      console.log(err);
+    }
+  }
+
+  await doc.destroy();
+
+  return {
+    numpages: doc.numPages,
+    numrender: counter,
+    info: metaData?.info,
+    metadata: metaData?.metadata,
+    text,
+    version,
+  };
+}
@@ -9,6 +9,7 @@ export { SimpleKVStore } from "./kvStore/SimpleKVStore";
 export * from "./kvStore/types";
 export { AstraDBVectorStore } from "./vectorStore/AstraDBVectorStore";
 export { MongoDBAtlasVectorSearch } from "./vectorStore/MongoDBAtlasVectorStore";
-export { SimpleVectorStore } from "./vectorStore/SimpleVectorStore";
 export { PGVectorStore } from "./vectorStore/PGVectorStore";
+export { PineconeVectorStore } from "./vectorStore/PineconeVectorStore";
+export { SimpleVectorStore } from "./vectorStore/SimpleVectorStore";
 export * from "./vectorStore/types";
@@ -16,6 +16,8 @@ export class PGVectorStore implements VectorStore {
  storesText: boolean = true;

  private collection: string = "";
+  private schemaName: string = PGVECTOR_SCHEMA;
+  private tableName: string = PGVECTOR_TABLE;

  /*
    FROM pg LIBRARY:
@@ -37,7 +39,10 @@ export class PGVectorStore implements VectorStore {
  */
  db?: pg.Client;

-  constructor() {}
+  constructor(config?: { schemaName?: string; tableName?: string }) {
+    this.schemaName = config?.schemaName ?? PGVECTOR_SCHEMA;
+    this.tableName = config?.tableName ?? PGVECTOR_TABLE;
+  }

  /**
   * Setter for the collection property.
@@ -66,7 +71,9 @@ export class PGVectorStore implements VectorStore {
      try {
        // Create DB connection
        // Read connection params from env - see comment block above
-        const db = new pg.Client();
+        const db = new pg.Client({
+          connectionString: process.env.PG_CONNECTION_STRING,
+        });
        await db.connect();

        // Check vector extension
@@ -88,9 +95,9 @@ export class PGVectorStore implements VectorStore {
  }

  private async checkSchema(db: pg.Client) {
-    await db.query(`CREATE SCHEMA IF NOT EXISTS ${PGVECTOR_SCHEMA}`);
+    await db.query(`CREATE SCHEMA IF NOT EXISTS ${this.schemaName}`);

-    const tbl = `CREATE TABLE IF NOT EXISTS ${PGVECTOR_SCHEMA}.${PGVECTOR_TABLE}(
+    const tbl = `CREATE TABLE IF NOT EXISTS ${this.schemaName}.${this.tableName}(
      id uuid DEFAULT gen_random_uuid() PRIMARY KEY,
      external_id VARCHAR,
      collection VARCHAR,
@@ -100,8 +107,8 @@ export class PGVectorStore implements VectorStore {
    )`;
    await db.query(tbl);

-    const idxs = `CREATE INDEX IF NOT EXISTS idx_${PGVECTOR_TABLE}_external_id ON ${PGVECTOR_SCHEMA}.${PGVECTOR_TABLE} (external_id);
-      CREATE INDEX IF NOT EXISTS idx_${PGVECTOR_TABLE}_collection ON ${PGVECTOR_SCHEMA}.${PGVECTOR_TABLE} (collection);`;
+    const idxs = `CREATE INDEX IF NOT EXISTS idx_${this.tableName}_external_id ON ${this.schemaName}.${this.tableName} (external_id);
+      CREATE INDEX IF NOT EXISTS idx_${this.tableName}_collection ON ${this.schemaName}.${this.tableName} (collection);`;
    await db.query(idxs);

    // TODO add IVFFlat or HNSW indexing?
@@ -126,7 +133,7 @@ export class PGVectorStore implements VectorStore {
   * @returns The result of the delete query.
   */
  async clearCollection() {
-    const sql: string = `DELETE FROM ${PGVECTOR_SCHEMA}.${PGVECTOR_TABLE} 
+    const sql: string = `DELETE FROM ${this.schemaName}.${this.tableName} 
      WHERE collection = $1`;

    const db = (await this.getDb()) as pg.Client;
@@ -147,7 +154,7 @@ export class PGVectorStore implements VectorStore {
      return Promise.resolve([]);
    }

-    const sql: string = `INSERT INTO ${PGVECTOR_SCHEMA}.${PGVECTOR_TABLE} 
+    const sql: string = `INSERT INTO ${this.schemaName}.${this.tableName} 
      (id, external_id, collection, document, metadata, embeddings) 
      VALUES ($1, $2, $3, $4, $5, $6)`;

@@ -197,7 +204,7 @@ export class PGVectorStore implements VectorStore {
    const collectionCriteria = this.collection.length
      ? "AND collection = $2"
      : "";
-    const sql: string = `DELETE FROM ${PGVECTOR_SCHEMA}.${PGVECTOR_TABLE} 
+    const sql: string = `DELETE FROM ${this.schemaName}.${this.tableName} 
      WHERE id = $1 ${collectionCriteria}`;

    const db = (await this.getDb()) as pg.Client;
@@ -230,7 +237,7 @@ export class PGVectorStore implements VectorStore {
    const sql = `SELECT 
        v.*, 
        embeddings <-> $1 s 
-      FROM ${PGVECTOR_SCHEMA}.${PGVECTOR_TABLE} v
+      FROM ${this.schemaName}.${this.tableName} v
      ${where}
      ORDER BY s 
      LIMIT ${max}
@@ -0,0 +1,220 @@
+import {
+  ExactMatchFilter,
+  MetadataFilters,
+  VectorStore,
+  VectorStoreQuery,
+  VectorStoreQueryResult,
+} from "./types";
+
+import { BaseNode, Document, Metadata, MetadataMode } from "../../Node";
+import { GenericFileSystem } from "../FileSystem";
+
+import {
+  FetchResponse,
+  Index,
+  Pinecone,
+  ScoredPineconeRecord,
+} from "@pinecone-database/pinecone";
+
+type PineconeParams = {
+  indexName?: string;
+  chunkSize?: number;
+};
+
+/**
+ * Provides support for writing and querying vector data in Postgres.
+ */
+export class PineconeVectorStore implements VectorStore {
+  storesText: boolean = true;
+
+  /*
+    FROM @pinecone-database/pinecone:
+      PINECONE_API_KEY="your_api_key"
+      PINECONE_ENVIRONMENT="your_environment"
+    Our addition:
+      PINECONE_INDEX_NAME="llama"
+      PINECONE_CHUNK_SIZE=100
+  */
+  db?: Pinecone;
+  indexName: string;
+  chunkSize: number;
+
+  constructor(params?: PineconeParams) {
+    this.indexName =
+      params?.indexName ?? process.env.PINECONE_INDEX_NAME ?? "llama";
+    this.chunkSize =
+      params?.chunkSize ??
+      Number.parseInt(process.env.PINECONE_CHUNK_SIZE ?? "100");
+  }
+
+  private async getDb(): Promise<Pinecone> {
+    if (!this.db) {
+      this.db = await new Pinecone();
+    }
+
+    return Promise.resolve(this.db);
+  }
+
+  /**
+   * Connects to the Pinecone account specified in environment vars.
+   * This method also checks and creates the named index if not found.
+   * @returns Pinecone client, or the error encountered while connecting/setting up.
+   */
+  client() {
+    return this.getDb();
+  }
+
+  async index() {
+    const db: Pinecone = await this.getDb();
+    return await db.index(this.indexName);
+  }
+
+  /**
+   * Delete all records for the current index.
+   * NOTE: This operation is not supported by Pinecone for "Starter" (free) indexes.
+   * @returns The result of the delete query.
+   */
+  async clearIndex() {
+    const db: Pinecone = await this.getDb();
+    return await db.index(this.indexName).deleteAll();
+  }
+
+  /**
+   * Adds vector record(s) to the table.
+   * @TODO Does not create or insert sparse vectors.
+   * @param embeddingResults The Nodes to be inserted, optionally including metadata tuples.
+   * @returns Due to limitations in the Pinecone client, does not return the upserted ID list, only a Promise resolve/reject.
+   */
+  async add(embeddingResults: BaseNode<Metadata>[]): Promise<string[]> {
+    if (embeddingResults.length == 0) {
+      return Promise.resolve([]);
+    }
+
+    const idx: Index = await this.index();
+    const nodes = embeddingResults.map(this.nodeToRecord);
+
+    for (let i = 0; i < nodes.length; i += this.chunkSize) {
+      const chunk = nodes.slice(i, i + this.chunkSize);
+      const result = await this.saveChunk(idx, chunk);
+      if (!result) {
+        return Promise.reject();
+      }
+    }
+    return Promise.resolve([]);
+  }
+
+  protected async saveChunk(idx: Index, chunk: any) {
+    try {
+      await idx.upsert(chunk);
+      return true;
+    } catch (err) {
+      const msg = `${err}`;
+      console.log(msg, err);
+      return false;
+    }
+  }
+
+  /**
+   * Deletes a single record from the database by id.
+   * NOTE: Uses the collection property controlled by setCollection/getCollection.
+   * @param refDocId Unique identifier for the record to delete.
+   * @param deleteKwargs Required by VectorStore interface.  Currently ignored.
+   * @returns Promise that resolves if the delete query did not throw an error.
+   */
+  async delete(refDocId: string, deleteKwargs?: any): Promise<void> {
+    const idx = await this.index();
+    return idx.deleteOne(refDocId);
+  }
+
+  /**
+   * Query the vector store for the closest matching data to the query embeddings
+   * @TODO QUERY TYPES
+   * @param query The VectorStoreQuery to be used
+   * @param options Required by VectorStore interface.  Currently ignored.
+   * @returns Zero or more Document instances with data from the vector store.
+   */
+  async query(
+    query: VectorStoreQuery,
+    options?: any,
+  ): Promise<VectorStoreQueryResult> {
+    const filter = this.toPineconeFilter(query.filters);
+
+    var options: any = {
+      vector: query.queryEmbedding,
+      topK: query.similarityTopK,
+      include_values: true,
+      include_metadara: true,
+      filter: filter,
+    };
+
+    const idx = await this.index();
+    const results = await idx.query(options);
+
+    const idList = results.matches.map((row) => row.id);
+    const records: FetchResponse<any> = await idx.fetch(idList);
+    const rows = Object.values(records.records);
+
+    const nodes = rows.map((row) => {
+      return new Document({
+        id_: row.id,
+        text: this.textFromResultRow(row),
+        metadata: this.metaWithoutText(row.metadata),
+        embedding: row.values,
+      });
+    });
+
+    const ret = {
+      nodes: nodes,
+      similarities: results.matches.map((row) => row.score || 999),
+      ids: results.matches.map((row) => row.id),
+    };
+
+    return Promise.resolve(ret);
+  }
+
+  /**
+   * Required by VectorStore interface.  Currently ignored.
+   * @param persistPath
+   * @param fs
+   * @returns Resolved Promise.
+   */
+  persist(
+    persistPath: string,
+    fs?: GenericFileSystem | undefined,
+  ): Promise<void> {
+    return Promise.resolve();
+  }
+
+  toPineconeFilter(stdFilters?: MetadataFilters) {
+    return stdFilters?.filters?.reduce((carry: any, item: ExactMatchFilter) => {
+      carry[item.key] = item.value;
+      return carry;
+    }, {});
+  }
+
+  textFromResultRow(row: ScoredPineconeRecord<Metadata>): string {
+    return row.metadata?.text ?? "";
+  }
+
+  metaWithoutText(meta: Metadata): any {
+    return Object.keys(meta)
+      .filter((key) => key != "text")
+      .reduce((acc: any, key: string) => {
+        acc[key] = meta[key];
+        return acc;
+      }, {});
+  }
+
+  nodeToRecord(node: BaseNode<Metadata>) {
+    let id: any = node.id_.length ? node.id_ : null;
+    let meta: any = node.metadata || {};
+    meta.create_date = new Date();
+    meta.text = node.getContent(MetadataMode.EMBED);
+
+    return {
+      id: id,
+      values: node.getEmbedding(),
+      metadata: meta,
+    };
+  }
+}
@@ -16,20 +16,17 @@ export function nodeToMetadata(
  textField: string = DEFAULT_TEXT_KEY,
  flatMetadata: boolean = false,
 ): Metadata {
-  const nodeObj = node.toJSON();
-  const metadata = node.metadata;
+  const { metadata, embedding, ...rest } = node.toMutableJSON();

  if (flatMetadata) {
-    validateIsFlat(node.metadata);
+    validateIsFlat(metadata);
  }

  if (removeText) {
-    nodeObj[textField] = "";
+    rest[textField] = "";
  }

-  nodeObj["embedding"] = null;
-
-  metadata["_node_content"] = JSON.stringify(nodeObj);
+  metadata["_node_content"] = JSON.stringify(rest);
  metadata["_node_type"] = node.constructor.name.replace("_", ""); // remove leading underscore to be compatible with Python

  metadata["document_id"] = node.sourceNode?.nodeId || "None";
@@ -40,17 +37,24 @@ export function nodeToMetadata(
 }

 export function metadataDictToNode(metadata: Metadata): BaseNode {
-  const nodeContent = metadata["_node_content"];
+  const {
+    _node_content: nodeContent,
+    _node_type: nodeType,
+    document_id,
+    doc_id,
+    ref_doc_id,
+    ...rest
+  } = metadata;
  if (!nodeContent) {
    throw new Error("Node content not found in metadata.");
  }
  const nodeObj = JSON.parse(nodeContent);
+  nodeObj.metadata = rest;

  // Note: we're using the name of the class stored in `_node_type`
  // and not the type attribute to reconstruct
  // the node. This way we're compatible with LlamaIndex Python
-  const node_type = metadata["_node_type"];
-  switch (node_type) {
+  switch (nodeType) {
    case "IndexNode":
      return jsonToNode(nodeObj, ObjectType.INDEX);
    default:
@@ -1,3 +1,5 @@
+import { Document } from "../Node";
+import { ServiceContext, serviceContextFromDefaults } from "../ServiceContext";
 import {
  CallbackManager,
  RetrievalCallbackResponse,
@@ -7,12 +9,7 @@ import { OpenAIEmbedding } from "../embeddings";
 import { SummaryIndex } from "../indices/summary";
 import { VectorStoreIndex } from "../indices/vectorStore/VectorStoreIndex";
 import { OpenAI } from "../llm/LLM";
-import { Document } from "../Node";
-import {
-  ResponseSynthesizer,
-  SimpleResponseBuilder,
-} from "../synthesizers";
-import { ServiceContext, serviceContextFromDefaults } from "../ServiceContext";
+import { ResponseSynthesizer, SimpleResponseBuilder } from "../synthesizers";
 import { mockEmbeddingModel, mockLlmGeneration } from "./utility/mockOpenAI";

 // Mock the OpenAI getOpenAISession function during testing
@@ -1,4 +1,4 @@
-import { SentenceSplitter, cjkSentenceTokenizer } from "../TextSplitter";
+import { cjkSentenceTokenizer, SentenceSplitter } from "../TextSplitter";

 describe("SentenceSplitter", () => {
  test("initializes", () => {
@@ -10,7 +10,7 @@ describe("SentenceSplitter", () => {
    const sentenceSplitter = new SentenceSplitter({});
    // generate the same line as above but correct syntax errors
    let splits = sentenceSplitter.getParagraphSplits(
-      "This is a paragraph.\n\nThis is another paragraph.",
+      "This is a paragraph.\n\n\nThis is another paragraph.",
      undefined,
    );
    expect(splits).toEqual([
@@ -88,7 +88,14 @@ describe("SentenceSplitter", () => {
      chunkingTokenizerFn: cjkSentenceTokenizer,
    });

-    const splits = sentenceSplitter.splitText("这是一个句子！这是另一个句子。");
-    expect(splits).toEqual(["这是一个句子！", "这是另一个句子。"]);
+    const splits = sentenceSplitter.splitText(
+      "此后如竟没有炬火：我便是唯一的光。倘若有了炬火，出了太阳，我们自然心悦诚服的消失。不但毫无不平，而且还要随喜赞美这炬火或太阳；因为他照了人类，连我都在内。",
+    );
+    expect(splits).toEqual([
+      "此后如竟没有炬火：我便是唯一的光。",
+      "倘若有了炬火，出了太阳，我们自然心悦诚服的消失。",
+      "不但毫无不平，而且还要随喜赞美这炬火或太阳；",
+      "因为他照了人类，连我都在内。",
+    ]);
  });
 });
@@ -0,0 +1,47 @@
+import { Document, MetadataMode } from "../Node";
+import {
+  metadataDictToNode,
+  nodeToMetadata,
+} from "../storage/vectorStore/utils";
+
+describe("Testing VectorStore utils", () => {
+  let node: Document;
+
+  beforeEach(() => {
+    node = new Document({
+      text: "text",
+      metadata: { meta1: "Some metadata" },
+    });
+  });
+
+  test("nodeToMetadata should not modify a node's metadata", () => {
+    nodeToMetadata(node, true);
+    expect(node.metadata).toEqual({ meta1: "Some metadata" });
+  });
+  test("metadataDictToNode should reconstructs node and remove text (except embedding)", () => {
+    const metadata = nodeToMetadata(node, true);
+    const newNode = metadataDictToNode(metadata);
+    expect(newNode.metadata).toEqual({ meta1: "Some metadata" });
+    expect(() => newNode.getEmbedding()).toThrow();
+    expect(newNode.getContent(MetadataMode.NONE)).toEqual("");
+  });
+  test("metadataDictToNode should reconstructs node (except embedding)", () => {
+    const metadata = nodeToMetadata(node, false);
+    const newNode = metadataDictToNode(metadata);
+    expect(newNode.metadata).toEqual({ meta1: "Some metadata" });
+    expect(newNode.getContent(MetadataMode.NONE)).toEqual("text");
+    expect(() => newNode.getEmbedding()).toThrow();
+  });
+  test("metadataDictToNode should not allow deep metadata if flatMetadata is true", () => {
+    node.metadata = { meta: { meta: "meta" } };
+    expect(() => nodeToMetadata(node, false, "text", true)).toThrow();
+  });
+  test("metadataDictToNode should throw an error when node content not found in metadata", () => {
+    const faultyMetadata = {
+      _node_type: "IndexNode",
+    };
+    expect(() => {
+      metadataDictToNode(faultyMetadata);
+    }).toThrow();
+  });
+});
@@ -8,18 +8,10 @@
    "preserveWatchOutput": true,
    "skipLibCheck": true,
    "strict": true,
-    "lib": [
-      "es2015",
-      "dom"
-    ],
+    "lib": ["es2015", "dom"],
    "target": "ES2015",
    "resolveJsonModule": true,
-    "typeRoots": [
-      "./types",
-      "./node_modules/@types"
-    ]
+    "typeRoots": ["./types", "./node_modules/@types"]
  },
-  "exclude": [
-    "node_modules"
-  ],
-}
+  "exclude": ["node_modules"]
+}
@@ -1,5 +1,13 @@
 # create-llama

+## 0.0.12
+
+### Patch Changes
+
+- 9c5e22a: Added cross-env so frontends with Express/FastAPI backends are working under Windows
+- 5ab65eb: Bring Python templates with TS templates to feature parity
+- 9c5e22a: Added vector DB selector to create-llama (starting with MongoDB support)
+
 ## 0.0.11

 ### Patch Changes
@@ -33,6 +33,7 @@ export async function createApp({
  model,
  communityProjectPath,
  vectorDb,
+  externalPort,
 }: InstallAppArgs): Promise<void> {
  const root = path.resolve(appPath);

@@ -73,6 +74,7 @@ export async function createApp({
    model,
    communityProjectPath,
    vectorDb,
+    externalPort,
  };

  if (frontend) {
@@ -87,7 +89,7 @@ export async function createApp({
      ...args,
      root: frontendRoot,
      framework: "nextjs",
-      customApiPath: "http://localhost:8000/api/chat",
+      customApiPath: `http://localhost:${externalPort ?? 8000}/api/chat`,
      backend: false,
    });
    // copy readme for fullstack
@@ -1,5 +1,8 @@
 /* eslint-disable turbo/no-undeclared-env-vars */
 import { expect, test } from "@playwright/test";
+import { ChildProcess } from "child_process";
+import fs from "fs";
+import path from "path";
 import type {
  TemplateEngine,
  TemplateFramework,
@@ -9,7 +12,11 @@ import type {
 import { createTestDir, runApp, runCreateLlama, type AppType } from "./utils";

 const templateTypes: TemplateType[] = ["streaming", "simple"];
-const templateFrameworks: TemplateFramework[] = ["nextjs", "express"];
+const templateFrameworks: TemplateFramework[] = [
+  "nextjs",
+  "express",
+  "fastapi",
+];
 const templateEngines: TemplateEngine[] = ["simple", "context"];
 const templateUIs: TemplateUI[] = ["shadcn", "html"];

@@ -31,30 +38,100 @@ for (const templateType of templateTypes) {
              ? "--no-frontend" // simple templates don't have frontends
              : "--frontend"
            : "";
-        test(`try create-llama ${templateType} ${templateFramework} ${templateEngine} ${templateUI} ${appType}`, async ({
-          page,
-        }) => {
-          const cwd = await createTestDir();
-          const name = runCreateLlama(
-            cwd,
-            templateType,
-            templateFramework,
-            templateEngine,
-            templateUI,
-            appType,
-          );
+        if (appType === "--no-frontend" && templateUI !== "html") {
+          // if there's no frontend, don't iterate over UIs
+          continue;
+        }
+        test.describe(`try create-llama ${templateType} ${templateFramework} ${templateEngine} ${templateUI} ${appType}`, async () => {
+          let port: number;
+          let externalPort: number;
+          let cwd: string;
+          let name: string;
+          let cps: ChildProcess[] = [];

-          const port = Math.floor(Math.random() * 10000) + 10000;
-          const cps = await runApp(cwd, name, appType, port);
+          test.beforeAll(async () => {
+            port = Math.floor(Math.random() * 10000) + 10000;
+            externalPort = port + 1;

-          // test frontend
-          if (appType !== "--no-frontend") {
+            cwd = await createTestDir();
+            name = runCreateLlama(
+              cwd,
+              templateType,
+              templateFramework,
+              templateEngine,
+              templateUI,
+              appType,
+              externalPort,
+            );
+
+            if (templateFramework !== "fastapi") {
+              // don't run the app for fastapi for now (adds python dependency)
+              cps = await runApp(cwd, name, appType, port, externalPort);
+            }
+          });
+
+          test("App folder should exist", async () => {
+            const dirExists = fs.existsSync(path.join(cwd, name));
+            expect(dirExists).toBeTruthy();
+          });
+          test("Frontend should have a title", async ({ page }) => {
+            test.skip(
+              appType === "--no-frontend" || templateFramework === "fastapi",
+            );
            await page.goto(`http://localhost:${port}`);
            await expect(page.getByText("Built by LlamaIndex")).toBeVisible();
-          }
-          // TODO: test backend using curl (would need OpenAI key)
+          });
+
+          test("Frontend should be able to submit a message and receive a response", async ({
+            page,
+          }) => {
+            test.skip(
+              appType === "--no-frontend" || templateFramework === "fastapi",
+            );
+            await page.goto(`http://localhost:${port}`);
+            await page.fill("form input", "hello");
+            await page.click("form button[type=submit]");
+            const response = await page.waitForResponse(
+              (res) => {
+                return res.url().includes("/api/chat") && res.status() === 200;
+              },
+              {
+                timeout: 1000 * 60,
+              },
+            );
+            const text = await response.text();
+            console.log("AI response when submitting message: ", text);
+            expect(response.ok()).toBeTruthy();
+          });
+
+          test("Backend should response when calling API", async ({
+            request,
+          }) => {
+            test.skip(
+              appType !== "--no-frontend" || templateFramework === "fastapi",
+            );
+            const response = await request.post(
+              `http://localhost:${port}/api/chat`,
+              {
+                data: {
+                  messages: [
+                    {
+                      role: "user",
+                      content: "Hello",
+                    },
+                  ],
+                },
+              },
+            );
+            const text = await response.text();
+            console.log("AI response when calling API: ", text);
+            expect(response.ok()).toBeTruthy();
+          });
+
          // clean processes
-          cps.forEach((cp) => cp.kill());
+          test.afterAll(async () => {
+            cps.map((cp) => cp.kill());
+          });
        });
      }
    }
@@ -12,6 +12,7 @@ export async function runApp(
  name: string,
  appType: AppType,
  port: number,
+  externalPort: number,
 ): Promise<ChildProcess[]> {
  const cps: ChildProcess[] = [];

@@ -22,7 +23,7 @@ export async function runApp(
          await createProcess(
            "npm run dev",
            path.join(cwd, name, "backend"),
-            port + 1,
+            externalPort,
          ),
        );
        cps.push(
@@ -71,6 +72,7 @@ export function runCreateLlama(
  templateEngine: string,
  templateUI: string,
  appType: AppType,
+  externalPort: number,
 ) {
  const createLlama = path.join(__dirname, "..", "dist", "index.js");

@@ -96,10 +98,12 @@ export function runCreateLlama(
    "--model",
    MODEL,
    "--open-ai-key",
-    "testKey",
+    process.env.OPENAI_API_KEY || "testKey",
    appType,
    "--eslint",
    "--use-npm",
+    "--external-port",
+    externalPort,
  ].join(" ");
  console.log(`running command '${command}' in ${cwd}`);
  execSync(command, {
@@ -106,14 +106,27 @@ const program = new Commander.Command(packageJson.name)
 `,
  )
  .option(
-    "--model",
+    "--model <model>",
    `

  Select OpenAI model to use. E.g. gpt-3.5-turbo.
+`,
+  )
+  .option(
+    "--external-port <external>",
+    `
+
+Select external port.
 `,
  )
  .allowUnknownOption()
  .parse(process.argv);
+if (process.argv.includes("--no-frontend")) {
+  program.frontend = false;
+}
+if (process.argv.includes("--no-eslint")) {
+  program.eslint = false;
+}

 const packageManager = !!program.useNpm
  ? "npm"
@@ -210,6 +223,7 @@ async function run(): Promise<void> {
    model: program.model,
    communityProjectPath: program.communityProjectPath,
    vectorDb: program.vectorDb,
+    externalPort: program.externalPort,
  });
  conf.set("preferences", preferences);
 }
@@ -1,6 +1,6 @@
 {
  "name": "create-llama",
-  "version": "0.0.11",
+  "version": "0.0.12",
  "keywords": [
    "rag",
    "llamaindex",
@@ -20,10 +20,11 @@
    "dist"
  ],
  "scripts": {
+    "clean": "rimraf --glob ./dist ./templates/**/__pycache__ ./templates/**/node_modules ./templates/**/poetry.lock",
    "dev": "ncc build ./index.ts -w -o dist/",
-    "build": "ncc build ./index.ts -o ./dist/ --minify --no-cache --no-source-map-register",
+    "build": "npm run clean && ncc build ./index.ts -o ./dist/ --minify --no-cache --no-source-map-register",
    "lint": "eslint . --ignore-pattern dist",
-    "e2e": "playwright test --reporter=list",
+    "e2e": "playwright test",
    "prepublishOnly": "cd ../../ && turbo run build"
  },
  "devDependencies": {
@@ -46,6 +47,8 @@
    "got": "10.7.0",
    "picocolors": "1.0.0",
    "prompts": "2.1.0",
+    "rimraf": "^5.0.5",
+    "smol-toml": "^1.1.3",
    "tar": "6.1.15",
    "terminal-link": "^3.0.0",
    "update-check": "1.5.4",
@@ -55,4 +58,4 @@
  "engines": {
    "node": ">=16.14.0"
  }
-}
+}
@@ -89,14 +89,8 @@ export const askQuestions = async (
        })),
        initial: 0,
      },
-      {
-        onCancel: () => {
-          console.error("Exiting.");
-          process.exit(1);
-        },
-      },
+      handlers,
    );
-
    program.communityProjectPath = communityProjectPath;
    preferences.communityProjectPath = communityProjectPath;
    return; // early return - no further questions needed for community projects
@@ -130,11 +124,12 @@ export const askQuestions = async (
    }
  }

-  if (program.framework === "express" || program.framework === "fastapi") {
-    if (process.argv.includes("--no-frontend")) {
-      program.frontend = false;
-    }
+  if (
+    program.template === "streaming" &&
+    (program.framework === "express" || program.framework === "fastapi")
+  ) {
    // if a backend-only framework is selected, ask whether we should create a frontend
+    // (only for streaming backends)
    if (program.frontend === undefined) {
      if (ciInfo.isCI) {
        program.frontend = getPrefOrDefault("frontend");
@@ -161,7 +156,6 @@ export const askQuestions = async (
      }
    }
  } else {
-    // single project if framework is nextjs
    program.frontend = false;
  }

@@ -189,59 +183,55 @@ export const askQuestions = async (
    }
  }

-  if (program.framework === "express" || program.framework === "nextjs") {
-    if (!program.model) {
-      if (ciInfo.isCI) {
-        program.model = getPrefOrDefault("model");
-      } else {
-        const { model } = await prompts(
-          {
-            type: "select",
-            name: "model",
-            message: "Which model would you like to use?",
-            choices: [
-              { title: "gpt-3.5-turbo", value: "gpt-3.5-turbo" },
-              { title: "gpt-4", value: "gpt-4" },
-              { title: "gpt-4-1106-preview", value: "gpt-4-1106-preview" },
-              {
-                title: "gpt-4-vision-preview",
-                value: "gpt-4-vision-preview",
-              },
-            ],
-            initial: 0,
-          },
-          handlers,
-        );
-        program.model = model;
-        preferences.model = model;
-      }
+  if (!program.model) {
+    if (ciInfo.isCI) {
+      program.model = getPrefOrDefault("model");
+    } else {
+      const { model } = await prompts(
+        {
+          type: "select",
+          name: "model",
+          message: "Which model would you like to use?",
+          choices: [
+            { title: "gpt-3.5-turbo", value: "gpt-3.5-turbo" },
+            { title: "gpt-4", value: "gpt-4" },
+            { title: "gpt-4-1106-preview", value: "gpt-4-1106-preview" },
+            {
+              title: "gpt-4-vision-preview",
+              value: "gpt-4-vision-preview",
+            },
+          ],
+          initial: 0,
+        },
+        handlers,
+      );
+      program.model = model;
+      preferences.model = model;
    }
  }

-  if (program.framework === "express" || program.framework === "nextjs") {
-    if (!program.engine) {
-      if (ciInfo.isCI) {
-        program.engine = getPrefOrDefault("engine");
-      } else {
-        const { engine } = await prompts(
-          {
-            type: "select",
-            name: "engine",
-            message: "Which data source would you like to use?",
-            choices: [
-              {
-                title: "No data, just a simple chat",
-                value: "simple",
-              },
-              { title: "Use an example PDF", value: "context" },
-            ],
-            initial: 1,
-          },
-          handlers,
-        );
-        program.engine = engine;
-        preferences.engine = engine;
-      }
+  if (!program.engine) {
+    if (ciInfo.isCI) {
+      program.engine = getPrefOrDefault("engine");
+    } else {
+      const { engine } = await prompts(
+        {
+          type: "select",
+          name: "engine",
+          message: "Which data source would you like to use?",
+          choices: [
+            {
+              title: "No data, just a simple chat",
+              value: "simple",
+            },
+            { title: "Use an example PDF", value: "context" },
+          ],
+          initial: 1,
+        },
+        handlers,
+      );
+      program.engine = engine;
+      preferences.engine = engine;
    }
    if (program.engine !== "simple" && !program.vectorDb) {
      if (ciInfo.isCI) {
@@ -282,11 +272,7 @@ export const askQuestions = async (
    preferences.openAiKey = key;
  }

-  if (
-    program.framework !== "fastapi" &&
-    !process.argv.includes("--eslint") &&
-    !process.argv.includes("--no-eslint")
-  ) {
+  if (program.framework !== "fastapi" && program.eslint === undefined) {
    if (ciInfo.isCI) {
      program.eslint = getPrefOrDefault("eslint");
    } else {
@@ -0,0 +1,36 @@
+/* eslint-disable turbo/no-undeclared-env-vars */
+import * as dotenv from "dotenv";
+import {
+  PGVectorStore,
+  SimpleDirectoryReader,
+  VectorStoreIndex,
+  storageContextFromDefaults,
+} from "llamaindex";
+import { STORAGE_DIR, checkRequiredEnvVars } from "./shared.mjs";
+
+dotenv.config();
+
+async function loadAndIndex() {
+  // load objects from storage and convert them into LlamaIndex Document objects
+  const documents = await new SimpleDirectoryReader().loadData({
+    directoryPath: STORAGE_DIR,
+  });
+
+  // create postgres vector store
+  const vectorStore = new PGVectorStore();
+  vectorStore.setCollection(STORAGE_DIR);
+  vectorStore.clearCollection();
+
+  // create index from all the Documents
+  console.log("Start creating embeddings...");
+  const storageContext = await storageContextFromDefaults({ vectorStore });
+  await VectorStoreIndex.fromDocuments(documents, { storageContext });
+  console.log(`Successfully created embeddings.`);
+}
+
+(async () => {
+  checkRequiredEnvVars();
+  await loadAndIndex();
+  console.log("Finished generating storage.");
+  process.exit(0);
+})();
@@ -0,0 +1,29 @@
+/* eslint-disable turbo/no-undeclared-env-vars */
+import {
+  ContextChatEngine,
+  LLM,
+  PGVectorStore,
+  VectorStoreIndex,
+  serviceContextFromDefaults,
+} from "llamaindex";
+import { CHUNK_OVERLAP, CHUNK_SIZE, checkRequiredEnvVars } from "./shared.mjs";
+
+async function getDataSource(llm: LLM) {
+  checkRequiredEnvVars();
+  const pgvs = new PGVectorStore();
+  const serviceContext = serviceContextFromDefaults({
+    llm,
+    chunkSize: CHUNK_SIZE,
+    chunkOverlap: CHUNK_OVERLAP,
+  });
+  return await VectorStoreIndex.fromVectorStore(pgvs, serviceContext);
+}
+
+export async function createChatEngine(llm: LLM) {
+  const index = await getDataSource(llm);
+  const retriever = index.asRetriever({ similarityTopK: 5 });
+  return new ContextChatEngine({
+    chatModel: llm,
+    retriever,
+  });
+}
@@ -0,0 +1,22 @@
+export const STORAGE_DIR = "./data";
+export const CHUNK_SIZE = 512;
+export const CHUNK_OVERLAP = 20;
+
+const REQUIRED_ENV_VARS = ["PG_CONNECTION_STRING", "OPENAI_API_KEY"];
+
+export function checkRequiredEnvVars() {
+  const missingEnvVars = REQUIRED_ENV_VARS.filter((envVar) => {
+    return !process.env[envVar];
+  });
+
+  if (missingEnvVars.length > 0) {
+    console.log(
+      `The following environment variables are required but missing: ${missingEnvVars.join(
+        ", ",
+      )}`,
+    );
+    throw new Error(
+      `Missing environment variables: ${missingEnvVars.join(", ")}`,
+    );
+  }
+}
@@ -0,0 +1,3 @@
+DATA_DIR = "data"  # directory containing the documents to index
+CHUNK_SIZE = 1024
+CHUNK_OVERLAP = 20
@@ -0,0 +1,14 @@
+from llama_index import ServiceContext
+
+from app.context import create_base_context
+from app.engine.constants import CHUNK_SIZE, CHUNK_OVERLAP
+
+
+def create_service_context():
+    base = create_base_context()
+    return ServiceContext.from_defaults(
+        llm=base.llm,
+        embed_model=base.embed_model,
+        chunk_size=CHUNK_SIZE,
+        chunk_overlap=CHUNK_OVERLAP,
+    )
@@ -0,0 +1,48 @@
+from dotenv import load_dotenv
+
+load_dotenv()
+import os
+import logging
+from llama_index.vector_stores import MongoDBAtlasVectorSearch
+
+from app.engine.constants import DATA_DIR
+from app.engine.context import create_service_context
+
+
+from llama_index import (
+    SimpleDirectoryReader,
+    VectorStoreIndex,
+    StorageContext,
+)
+
+logging.basicConfig(level=logging.INFO)
+logger = logging.getLogger()
+
+
+def generate_datasource(service_context):
+    logger.info("Creating new index")
+    # load the documents and create the index
+    documents = SimpleDirectoryReader(DATA_DIR).load_data()
+    store = MongoDBAtlasVectorSearch(
+        db_name=os.environ["MONGODB_DATABASE"],
+        collection_name=os.environ["MONGODB_VECTORS"],
+        index_name=os.environ["MONGODB_VECTOR_INDEX"],
+    )
+    storage_context = StorageContext.from_defaults(vector_store=store)
+    VectorStoreIndex.from_documents(
+        documents,
+        service_context=service_context,
+        storage_context=storage_context,
+        show_progress=True,  # this will show you a progress bar as the embeddings are created
+    )
+    logger.info(
+        f"Successfully created embeddings in the MongoDB collection {os.environ['MONGODB_VECTORS']}"
+    )
+    logger.info(
+        """IMPORTANT: You can't query your index yet because you need to create a vector search index in MongoDB's UI now.
+See https://github.com/run-llama/mongodb-demo/tree/main?tab=readme-ov-file#create-a-vector-search-index"""
+    )
+
+
+if __name__ == "__main__":
+    generate_datasource(create_service_context())
@@ -0,0 +1,23 @@
+import logging
+import os
+
+from llama_index import (
+    VectorStoreIndex,
+)
+from llama_index.vector_stores import MongoDBAtlasVectorSearch
+
+from app.engine.context import create_service_context
+
+
+def get_chat_engine():
+    service_context = create_service_context()
+    logger = logging.getLogger("uvicorn")
+    logger.info("Connecting to index from MongoDB...")
+    store = MongoDBAtlasVectorSearch(
+        db_name=os.environ["MONGODB_DATABASE"],
+        collection_name=os.environ["MONGODB_VECTORS"],
+        index_name=os.environ["MONGODB_VECTOR_INDEX"],
+    )
+    index = VectorStoreIndex.from_vector_store(store, service_context)
+    logger.info("Finished connecting to index from MongoDB.")
+    return index.as_chat_engine(similarity_top_k=5)
@@ -0,0 +1,4 @@
+STORAGE_DIR = "storage"  # directory to cache the generated index
+DATA_DIR = "data"  # directory containing the documents to index
+CHUNK_SIZE = 1024
+CHUNK_OVERLAP = 20
@@ -0,0 +1,14 @@
+from llama_index import ServiceContext
+
+from app.context import create_base_context
+from app.engine.constants import CHUNK_SIZE, CHUNK_OVERLAP
+
+
+def create_service_context():
+    base = create_base_context()
+    return ServiceContext.from_defaults(
+        llm=base.llm,
+        embed_model=base.embed_model,
+        chunk_size=CHUNK_SIZE,
+        chunk_overlap=CHUNK_OVERLAP,
+    )
@@ -0,0 +1,31 @@
+import logging
+
+from dotenv import load_dotenv
+
+from app.engine.constants import DATA_DIR, STORAGE_DIR
+from app.engine.context import create_service_context
+
+load_dotenv()
+
+from llama_index import (
+    SimpleDirectoryReader,
+    VectorStoreIndex,
+)
+
+logging.basicConfig(level=logging.INFO)
+logger = logging.getLogger()
+
+
+def generate_datasource(service_context):
+    logger.info("Creating new index")
+    # load the documents and create the index
+    documents = SimpleDirectoryReader(DATA_DIR).load_data()
+    index = VectorStoreIndex.from_documents(documents, service_context=service_context)
+    # store it for later
+    index.storage_context.persist(STORAGE_DIR)
+    logger.info(f"Finished creating new index. Stored in {STORAGE_DIR}")
+
+
+if __name__ == "__main__":
+    service_context = create_service_context()
+    generate_datasource(service_context)
@@ -0,0 +1,25 @@
+import logging
+import os
+from llama_index import (
+    StorageContext,
+    load_index_from_storage,
+)
+
+from app.engine.constants import STORAGE_DIR
+from app.engine.context import create_service_context
+
+
+def get_chat_engine():
+    service_context = create_service_context()
+    # check if storage already exists
+    if not os.path.exists(STORAGE_DIR):
+        raise Exception(
+            "StorageContext is empty - call 'python app/engine/generate.py' to generate the storage first"
+        )
+    logger = logging.getLogger("uvicorn")
+    # load the existing index
+    logger.info(f"Loading index from {STORAGE_DIR}...")
+    storage_context = StorageContext.from_defaults(persist_dir=STORAGE_DIR)
+    index = load_index_from_storage(storage_context, service_context=service_context)
+    logger.info(f"Finished loading index from {STORAGE_DIR}")
+    return index.as_chat_engine()
@@ -11,7 +11,7 @@ import { STORAGE_DIR, checkRequiredEnvVars } from "./shared.mjs";

 dotenv.config();

-const mongoUri = process.env.MONGODB_URI;
+const mongoUri = process.env.MONGO_URI;
 const databaseName = process.env.MONGODB_DATABASE;
 const vectorCollectionName = process.env.MONGODB_VECTORS;
 const indexName = process.env.MONGODB_VECTOR_INDEX;
@@ -11,7 +11,7 @@ import { checkRequiredEnvVars, CHUNK_OVERLAP, CHUNK_SIZE } from "./shared.mjs";

 async function getDataSource(llm: LLM) {
  checkRequiredEnvVars();
-  const client = new MongoClient(process.env.MONGODB_URI!);
+  const client = new MongoClient(process.env.MONGO_URI!);
  const serviceContext = serviceContextFromDefaults({
    llm,
    chunkSize: CHUNK_SIZE,
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
thucpn	4a520e05ff	Merge branch 'feat/integrate-postgresql-datasource' of github.com:run-llama/LlamaIndexTS into feat/integrate-postgresql-datasource	2024-01-04 15:37:12 +07:00
thucpn	b6c109baa5	chore: resolve conflict	2024-01-04 15:36:58 +07:00
Thuc Pham	a89534b704	feat: use connection string for create llama	2024-01-04 15:36:21 +07:00
Thuc Pham	9132d3c23e	feat: integrate create-llama with postgresql on cloud	2024-01-04 15:36:21 +07:00
Thuc Pham	e78eece82d	feat: update document to use timescale postgresql	2024-01-04 15:36:20 +07:00
thucpn	29b0f078ed	fix: add await to connect db	2024-01-04 15:36:20 +07:00
thucpn	4ba9a6cbe5	docs: update question and log	2024-01-04 15:36:20 +07:00
thucpn	0e4290e525	feat: integrate postgresql datasource	2024-01-04 15:35:44 +07:00
thucpn	1ea126bb5b	chore: resolve conflict	2024-01-04 15:33:02 +07:00
Thuc Pham	426eb43b41	feat: use connection string for create llama	2024-01-04 07:47:03 +00:00
Marcus Schiesser	f85c042a94	refactor: encapsulate node serialization	2024-01-04 10:19:18 +07:00
yisding	40e892f813	Oops adding back Alex's crypto changes	2024-01-04 10:19:18 +07:00
yisding	2381025306	move clone outside of toJSON I like structuredClone although we have an issue with AWS only supporting Node 16	2024-01-04 10:19:18 +07:00
Marcus Schiesser	a0909dc053	docs: updated how to create a mongodb vector index	2024-01-04 10:19:18 +07:00
Marcus Schiesser	853a14b7c7	fix: add test cases, ensure that a node's metadata is not modified and remove text if requested	2024-01-04 10:19:18 +07:00
yisding	b31e2d42eb	fixed recursive issue in metadataDictToNode	2024-01-04 10:19:18 +07:00
yisding	5b6ad9419f	fix mongo example	2024-01-04 10:19:18 +07:00
Thuc Pham	47f706bbd5	feat: integrate create-llama with postgresql on cloud	2024-01-03 10:29:41 +00:00
Thuc Pham	55fc9c6beb	feat: update document to use timescale postgresql	2024-01-03 09:13:42 +00:00
Marcus Schiesser	a60948a87e	fix: add format check to pre-commit	2024-01-03 10:39:54 +07:00
Alex Yang	8b420da753	style: prettier format (#304 )	2024-01-02 17:12:06 -06:00
Alex Yang	12c079b74a	refactor: remove unused deps (#303 )	2024-01-02 16:40:24 -06:00
Marcus Schiesser	c835f78dd0	docs: added changesets for core	2024-01-02 16:07:12 +07:00
Marcus Schiesser	d52eb9d4ee	fix[docs]: installation link (#302 )	2024-01-02 12:25:56 +08:00
Marcus Schiesser	0cfd9f60b5	create-llama@0.0.12	2024-01-02 11:02:34 +07:00
Marcus Schiesser	5ab65eb95a	fix[cl]: naming bug and added release changeset	2023-12-29 17:58:43 +07:00
Marcus Schiesser	b13fb36de0	Merge pull request #295 from run-llama/ms/cl-python-add-mongodb Feat: Add MongoDB Vector DB support for Python projects in create-llama	2023-12-29 15:01:06 +07:00
Marcus Schiesser	d8fe65a273	refactor: improved var naming	2023-12-29 14:26:29 +07:00
Marcus Schiesser	ba1cb996cf	refactor: separate python and typescript generators	2023-12-29 14:10:01 +07:00
thucpn	41f41d6543	feat: prepare python dependencies	2023-12-29 13:47:41 +07:00
thucpn	e25fc44db9	Merge branch 'ms/cl-python-add-mongodb' of github.com:run-llama/LlamaIndexTS into ms/cl-python-add-mongodb	2023-12-28 14:53:22 +07:00
thucpn	2ea91dc94b	docs: update packages	2023-12-28 14:52:12 +07:00
Marcus Schiesser	b16419ad3e	Update packages/create-llama/templates/index.ts	2023-12-28 15:12:03 +08:00
thucpn	1e3c05c408	feat: question to select vectordb for python template	2023-12-28 14:03:35 +07:00
Marcus Schiesser	a7eb59f472	fix: update pnpm-lock.yaml	2023-12-28 13:58:11 +07:00
Marcus Schiesser	e99448481c	Merge pull request #296 from run-llama/add-pinecone-vector-store Add pinecone vector store	2023-12-28 13:49:40 +07:00
Michael Tutty	83ab7622d9	Fix reference in examples/pinecone-vector-store	2023-12-28 13:44:35 +07:00
Michael Tutty	d4312d504b	Resolve PR issues for pinecone-vector-store	2023-12-28 13:44:35 +07:00
Michael Tutty	95742e7704	AddPineconeVectorStore to storage/index.ts	2023-12-28 13:44:35 +07:00
Michael Tutty	a8845a33df	Add apps/simple/pinecone-vector-store	2023-12-28 13:44:35 +07:00
Michael Tutty	b3fd87f302	Add PineconeVectorStore	2023-12-28 13:44:34 +07:00
Marcus Schiesser	25ba970e09	feat: added python code for mongodb	2023-12-28 11:47:41 +07:00
Marcus Schiesser	a67f9aaad7	Merge pull request #292 from run-llama/ms/cl-python-features Feat: Bring Python templates with TS templates to feature parity	2023-12-28 11:31:11 +07:00
Marcus Schiesser	210ce35867	fix: remove cleaning the build assets (doesn't work as due to how ncc references the assets)	2023-12-28 10:38:21 +07:00
Marcus Schiesser	36905f6442	fix: CI not running on windows	2023-12-27 18:38:35 +07:00
Marcus Schiesser	ed509db04a	feat[e2e]: add simple check for fastapi (folder exists)	2023-12-27 18:05:44 +07:00
Marcus Schiesser	14413c0637	fix: produce clean create-llama builds	2023-12-27 17:59:29 +07:00
Marcus Schiesser	9682c95da8	fix: incorrect generation message	2023-12-27 15:21:38 +07:00
Marcus Schiesser	7c6eba90e5	fix: don't allow frontend for non-streaming	2023-12-27 15:21:38 +07:00
Marcus Schiesser	c85bf225b9	fix: python packaging	2023-12-27 15:21:36 +07:00
Marcus Schiesser	b51c2d66a5	fix: get embed_model from base model	2023-12-27 15:20:45 +07:00
Marcus Schiesser	935bc52239	fix: use base service context	2023-12-27 15:20:45 +07:00
thucpn	57ff51823c	docs: update readme for simple template	2023-12-27 15:20:45 +07:00
thucpn	06b20a1772	fix: code review and bugs	2023-12-27 15:20:45 +07:00
thucpn	7a98255149	fix: path to typescript folder	2023-12-27 15:20:45 +07:00
thucpn	980038c711	refactor: typescript vectordb folder	2023-12-27 15:20:45 +07:00
thucpn	09d4e36200	feat: create chat engine folder for python	2023-12-27 15:20:45 +07:00
thucpn	8fb523bcef	fix: use model env for all framework	2023-12-27 15:20:45 +07:00
thucpn	e48a621d61	fix: use public model only for nextjs	2023-12-27 15:20:45 +07:00
thucpn	1f87787b05	refactor: context structure for simple python template	2023-12-27 15:20:45 +07:00
thucpn	fb6cef8a0b	docs: update env config	2023-12-27 15:20:45 +07:00
thucpn	21368f6218	feat: remove constants.ts in ts templates	2023-12-27 15:20:45 +07:00
Marcus Schiesser	116685017d	feat[cl-fastapi]: test and document new fastapi structure	2023-12-27 15:20:45 +07:00
Marcus Schiesser	23082f2c5e	feat[cl-fastapi]: draft for new fastapi structure (supporting engines)	2023-12-27 15:20:45 +07:00
Marcus Schiesser	ff2b3ca727	fix[cl-fastapi]: use json for request content-type (and update llama-index)	2023-12-27 15:20:45 +07:00
thucpn	9a42038a71	fix: add await to connect db	2023-12-26 17:23:24 +07:00
thucpn	c4b43e5afb	docs: update question and log	2023-12-26 17:10:19 +07:00
Marcus Schiesser	e55d41f5df	fix: don't include python caches to npm	2023-12-22 16:22:34 +07:00
Marcus Schiesser	91fb5101d6	refactor: don't check in idea projects	2023-12-22 15:34:09 +07:00
Marcus Schiesser	9c5e22a656	docs: added changesets for create-llama	2023-12-22 15:34:09 +07:00
Marcus Schiesser	18f23b298e	feat: add /api/chat e2e test (uses openai key) (#287 ) * feat: allow custom external port --------- Co-authored-by: thucpn <thucsh2@gmail.com>	2023-12-22 12:49:13 +07:00
Alex Yang	ddf39ebeaa	refactor: sentence split (#290 ) Co-authored-by: Marcus Schiesser <mail@marcusschiesser.de>	2023-12-21 12:36:43 -06:00
thucpn	03e302f6b1	feat: integrate postgresql datasource	2023-12-21 11:10:37 +07:00
Alex Yang	320b515e7d	fix: align separator with llama_index (#289 )	2023-12-20 13:15:14 -06:00
Marcus Schiesser	04c50ee946	Feat: Removed pdf-parse, and directly use latest pdf.js (#288 )	2023-12-20 17:13:26 +08:00