rolled back notion package and changeset

Merge pull request #112 from kkang2097/fix-output-parser
Create OutputParser.test.ts [Needs Merge]
2026-07-15 06:52:45 -04:00 · 2023-09-14 10:41:00 -07:00 · 2023-09-14 09:32:35 -07:00 · 2023-09-13 20:49:22 -07:00 · 2023-09-13 18:49:11 -07:00 · 2023-09-13 15:30:55 -07:00
90 changed files with 2545 additions and 832 deletions
@@ -1,5 +0,0 @@
---
-"llamaindex": patch
---
-
-Renamed ListIndex to SummaryIndex to better indicate its use.
@@ -1,5 +0,0 @@
---
-"llamaindex": patch
---
-
-Strong types for prompts.
@@ -0,0 +1,5 @@
+---
+"llamaindex": patch
+---
+
+Typesafe metadata (thanks @TomPenguin)
@@ -0,0 +1,5 @@
+---
+"llamaindex": patch
+---
+
+MongoReader (thanks @kkang2097)
@@ -0,0 +1,5 @@
+---
+"llamaindex": patch
+---
+
+Make OutputParser less strict and add tests (Thanks @kkang2097)
@@ -1,5 +1,5 @@
 name: Bugfix
-title: 'Sweep: '
+title: "Sweep: "
 description: Write something like "We notice ... behavior when ... happens instead of ...""
 labels: sweep
 body:
@@ -8,4 +8,4 @@ body:
    attributes:
      label: Details
      description: More details about the bug
-      placeholder: The bug might be in ... file
+      placeholder: The bug might be in ... file
@@ -1,5 +1,5 @@
 name: Feature Request
-title: 'Sweep: '
+title: "Sweep: "
 description: Write something like "Write an api endpoint that does "..." in the "..." file"
 labels: sweep
 body:
@@ -8,4 +8,4 @@ body:
    attributes:
      label: Details
      description: More details for Sweep
-      placeholder: The new endpoint should use the ... class from ... file because it contains ... logic
+      placeholder: The new endpoint should use the ... class from ... file because it contains ... logic
@@ -1,5 +1,5 @@
 name: Refactor
-title: 'Sweep: '
+title: "Sweep: "
 description: Write something like "Modify the ... api endpoint to use ... version and ... framework"
 labels: sweep
 body:
@@ -8,4 +8,4 @@ body:
    attributes:
      label: Details
      description: More details for Sweep
-      placeholder: We are migrating this function to ... version because ...
+      placeholder: We are migrating this function to ... version because ...
@@ -22,4 +22,4 @@ jobs:
        run: pnpm install

      - name: Run lint
-        run: pnpm run lint
+        run: pnpm run lint
@@ -7,18 +7,18 @@ jobs:
    runs-on: ubuntu-latest

    steps:
-    - name: Checkout code
-      uses: actions/checkout@v2
+      - name: Checkout code
+        uses: actions/checkout@v2

-    - name: Setup Node.js
-      uses: actions/setup-node@v2
-      with:
-        node-version: '18'
+      - name: Setup Node.js
+        uses: actions/setup-node@v2
+        with:
+          node-version: "18"

-    - name: Install dependencies
-      run: |
-        npm i -g pnpm
-        pnpm install
+      - name: Install dependencies
+        run: |
+          npm i -g pnpm
+          pnpm install

-    - name: Run tests
-      run: pnpm run test
+      - name: Run tests
+        run: pnpm run test
@@ -20,7 +20,7 @@ In a new folder:
 export OPENAI_API_KEY="sk-......" # Replace with your key from https://platform.openai.com/account/api-keys
 pnpm init
 pnpm install typescript
-pnpm exec tsc –-init # if needed
+pnpm exec tsc --init # if needed
 pnpm install llamaindex
 pnpm install @types/node
 ```
@@ -36,7 +36,7 @@ async function main() {
  // Load essay from abramov.txt in Node
  const essay = await fs.readFile(
    "node_modules/llamaindex/examples/abramov.txt",
-    "utf-8"
+    "utf-8",
  );

  // Create Document object with essay
@@ -48,7 +48,7 @@ async function main() {
  // Query the index
  const queryEngine = index.asQueryEngine();
  const response = await queryEngine.query(
-    "What did the author do in college?"
+    "What did the author do in college?",
  );

  // Output response
@@ -61,7 +61,7 @@ main();
 Then you can run it using

 ```bash
-pnpm dlx ts-node example.ts
+pnpx ts-node example.ts
 ```

 ## Playground
@@ -1,3 +1,3 @@
 module.exports = {
-  presets: [require.resolve('@docusaurus/core/lib/babel/preset')],
+  presets: [require.resolve("@docusaurus/core/lib/babel/preset")],
 };
@@ -8,40 +8,42 @@ LlamaIndex.TS helps you build LLM-powered applications (e.g. Q&A, chatbot) over

 In this high-level concepts guide, you will learn:

-* how an LLM can answer questions using your own data.
-* key concepts and modules in LlamaIndex.TS for composing your own query pipeline.
+- how an LLM can answer questions using your own data.
+- key concepts and modules in LlamaIndex.TS for composing your own query pipeline.

 ## Answering Questions Across Your Data

 LlamaIndex uses a two stage method when using an LLM with your data:

-1) **indexing stage**: preparing a knowledge base, and
-2) **querying stage**: retrieving relevant context from the knowledge to assist the LLM in responding to a question
+1. **indexing stage**: preparing a knowledge base, and
+2. **querying stage**: retrieving relevant context from the knowledge to assist the LLM in responding to a question

 ![](./_static/concepts/rag.jpg)

 This process is also known as Retrieval Augmented Generation (RAG).

-LlamaIndex.TS provides the essential toolkit for making both steps super easy. 
+LlamaIndex.TS provides the essential toolkit for making both steps super easy.

 Let's explore each stage in detail.

 ### Indexing Stage
+
 LlamaIndex.TS help you prepare the knowledge base with a suite of data connectors and indexes.

-![](./_static/concepts/indexing.jpg) 
+![](./_static/concepts/indexing.jpg)

 [**Data Loaders**](./modules/high_level/data_loader.md):
 A data connector (i.e. `Reader`) ingest data from different data sources and data formats into a simple `Document` representation (text and simple metadata).

 [**Documents / Nodes**](./modules/high_level/documents_and_nodes.md): A `Document` is a generic container around any data source - for instance, a PDF, an API output, or retrieved data from a database. A `Node` is the atomic unit of data in LlamaIndex and represents a "chunk" of a source `Document`. It's a rich representation that includes metadata and relationships (to other nodes) to enable accurate and expressive retrieval operations.

-[**Data Indexes**](./modules/high_level/data_index.md): 
+[**Data Indexes**](./modules/high_level/data_index.md):
 Once you've ingested your data, LlamaIndex helps you index data into a format that's easy to retrieve.

 Under the hood, LlamaIndex parses the raw documents into intermediate representations, calculates vector embeddings, and stores your data in-memory or to disk.

 ### Querying Stage
+
 In the querying stage, the query pipeline retrieves the most relevant context given a user query,
 and pass that to the LLM (along with the query) to synthesize a response.

@@ -57,12 +59,13 @@ These building blocks can be customized to reflect ranking preferences, as well
 ![](./_static/concepts/querying.jpg)

 #### Building Blocks
-[**Retrievers**](./modules/low_level/retriever.md): 
+
+[**Retrievers**](./modules/low_level/retriever.md):
 A retriever defines how to efficiently retrieve relevant context from a knowledge base (i.e. index) when given a query.
 The specific retrieval logic differs for difference indices, the most popular being dense retrieval against a vector index.

 [**Response Synthesizers**](./modules/low_level/response_synthesizer.md):
-A response synthesizer generates a response from an LLM, using a user query and a given set of retrieved text chunks.  
+A response synthesizer generates a response from an LLM, using a user query and a given set of retrieved text chunks.

 #### Pipelines

@@ -70,7 +73,6 @@ A response synthesizer generates a response from an LLM, using a user query and
 A query engine is an end-to-end pipeline that allow you to ask question over your data.
 It takes in a natural language query, and returns a response, along with reference context retrieved and passed to the LLM.

-
-[**Chat Engines**](./modules/high_level/chat_engine.md): 
+[**Chat Engines**](./modules/high_level/chat_engine.md):
 A chat engine is an end-to-end pipeline for having a conversation with your data
 (multiple back-and-forth instead of a single question & answer).
@@ -10,14 +10,14 @@ We include several end-to-end examples using LlamaIndex.TS in the repository

 Read a file and chat about it with the LLM.

-## [List Index](https://github.com/run-llama/LlamaIndexTS/blob/main/apps/simple/listIndex.ts)
-
-Create a list index and query it. This example also use the `LLMRetriever`, which will use the LLM to select the best nodes to use when generating answer.
-
 ## [Vector Index](https://github.com/run-llama/LlamaIndexTS/blob/main/apps/simple/vectorIndex.ts)

 Create a vector index and query it. The vector index will use embeddings to fetch the top k most relevant nodes. By default, the top k is 2.

+## [Summary Index](https://github.com/run-llama/LlamaIndexTS/blob/main/apps/simple/summaryIndex.ts)
+
+Create a list index and query it. This example also use the `LLMRetriever`, which will use the LLM to select the best nodes to use when generating answer.
+
 ## [Save / Load an Index](https://github.com/run-llama/LlamaIndexTS/blob/main/apps/simple/storageContext.ts)

 Create and load a vector index. Persistance to disk in LlamaIndex.TS happens automatically once a storage context object is created.
@@ -28,7 +28,7 @@ Create a vector index and query it, while also configuring the the `LLM`, the `S

 ## [OpenAI LLM](https://github.com/run-llama/LlamaIndexTS/blob/main/apps/simple/openai.ts)

-Create an OpenAI LLM and directly use it for chat. 
+Create an OpenAI LLM and directly use it for chat.

 ## [Llama2 DeuceLLM](https://github.com/run-llama/LlamaIndexTS/blob/main/apps/simple/llamadeuce.ts)

@@ -40,4 +40,4 @@ Uses the `SubQuestionQueryEngine`, which breaks complex queries into multiple qu

 ## [Low Level Modules](https://github.com/run-llama/LlamaIndexTS/blob/main/apps/simple/lowlevel.ts)

-This example uses several low-level components, which removes the need for an actual query engine. These components can be used anywhere, in any application, or customized and sub-classed to meet your own needs.
+This example uses several low-level components, which removes the need for an actual query engine. These components can be used anywhere, in any application, or customized and sub-classed to meet your own needs.
@@ -1,3 +1,3 @@
 label: "Modules"
 collapsed: false
-position: 5
+position: 5
@@ -1 +1 @@
-label: High-Level Modules
+label: High-Level Modules
@@ -6,23 +6,18 @@ sidebar_position: 2

 An index is the basic container and organization for your data. LlamaIndex.TS supports two indexes:

- `ListIndex` - will send every `Node` in the index to the LLM in order to generate a response
 - `VectorStoreIndex` - will send the top-k `Node`s to the LLM when generating a response. The default top-k is 2.
+- `SummaryIndex` - will send every `Node` in the index to the LLM in order to generate a response

 ```typescript
-import {
-  Document,
-  VectorStoreIndex,
-} from "llamaindex";
+import { Document, VectorStoreIndex } from "llamaindex";

 const document = new Document({ text: "test" });

-const index = await VectorStoreIndex.fromDocuments(
-  [document]
-);
+const index = await VectorStoreIndex.fromDocuments([document]);
 ```

 ## API Reference

- [ListIndex](../../api/classes/ListIndex.md)
- [VectorStoreIndex](../../api/classes/VectorStoreIndex.md)
+- [SummaryIndex](../../api/classes/SummaryIndex.md)
+- [VectorStoreIndex](../../api/classes/VectorStoreIndex.md)
@@ -9,7 +9,7 @@ sidebar_position: 0
 ```typescript
 import { Document } from "llamaindex";

-document = new Document({ text: "text", metadata: { "key": "val" }});
+document = new Document({ text: "text", metadata: { key: "val" } });
 ```

 ## API Reference
@@ -1 +1 @@
-label: Low-Level Modules
+label: Low-Level Modules
@@ -4,7 +4,7 @@ sidebar_position: 1

 # Embedding

-The embedding model in LlamaIndex is responsible for creating numerical representations of text. By default, LlamaIndex will use the `text-embedding-ada-002` model from OpenAI. 
+The embedding model in LlamaIndex is responsible for creating numerical representations of text. By default, LlamaIndex will use the `text-embedding-ada-002` model from OpenAI.

 This can be explicitly set in the `ServiceContext` object.

@@ -4,7 +4,7 @@ sidebar_position: 0

 # LLM

-The LLM is responsible for reading text and generating natural language responses to queries. By default, LlamaIndex.TS uses `gpt-3.5-turbo`. 
+The LLM is responsible for reading text and generating natural language responses to queries. By default, LlamaIndex.TS uses `gpt-3.5-turbo`.

 The LLM can be explicitly set in the `ServiceContext` object.

@@ -19,4 +19,4 @@ const serviceContext = serviceContextFromDefaults({ llm: openaiLLM });
 ## API Reference

 - [OpenAI](../../api/classes/OpenAI.md)
- [ServiceContext](../../api/interfaces/ServiceContext.md)
+- [ServiceContext](../../api/interfaces/ServiceContext.md)
@@ -7,10 +7,7 @@ sidebar_position: 3
 The `NodeParser` in LlamaIndex is responbile for splitting `Document` objects into more manageable `Node` objects. When you call `.fromDocuments()`, the `NodeParser` from the `ServiceContext` is used to do this automatically for you. Alternatively, you can use it to split documents ahead of time.

 ```typescript
-import {
-  Document,
-  SimpleNodeParser,
-} from "llamaindex";
+import { Document, SimpleNodeParser } from "llamaindex";

 const nodeParser = new SimpleNodeParser();
 const nodes = nodeParser.getNodesFromDocuments([
@@ -25,7 +22,7 @@ The underlying text splitter will split text by sentences. It can also be used a
 ```typescript
 import { SentenceSplitter } from "llamaindex";

-const splitter = new SentenceSplitter({ chunkSize: 1, });
+const splitter = new SentenceSplitter({ chunkSize: 1 });

 const textSplits = splitter.splitText("Hello World");
 ```
@@ -6,26 +6,21 @@ sidebar_position: 6

 The ResponseSynthesizer is responsible for sending the query, nodes, and prompt templates to the LLM to generate a response. There are a few key modes for generating a response:

- `Refine`: "create and refine" an answer by sequentially going through each retrieved text chunk. 
-    This makes a separate LLM call per Node. Good for more detailed answers.
- `CompactAndRefine` (default): "compact" the prompt during each LLM call by stuffing as 
-    many text chunks that can fit within the maximum prompt size. If there are 
-    too many chunks to stuff in one prompt, "create and refine" an answer by going through
-    multiple compact prompts. The same as `refine`, but should result in less LLM calls.
- `TreeSummarize`: Given a set of text chunks and the query, recursively construct a tree 
-    and return the root node as the response. Good for summarization purposes.
+- `Refine`: "create and refine" an answer by sequentially going through each retrieved text chunk.
+  This makes a separate LLM call per Node. Good for more detailed answers.
+- `CompactAndRefine` (default): "compact" the prompt during each LLM call by stuffing as
+  many text chunks that can fit within the maximum prompt size. If there are
+  too many chunks to stuff in one prompt, "create and refine" an answer by going through
+  multiple compact prompts. The same as `refine`, but should result in less LLM calls.
+- `TreeSummarize`: Given a set of text chunks and the query, recursively construct a tree
+  and return the root node as the response. Good for summarization purposes.
 - `SimpleResponseBuilder`: Given a set of text chunks and the query, apply the query to each text
-    chunk while accumulating the responses into an array. Returns a concatenated string of all
-    responses. Good for when you need to run the same query separately against each text
-    chunk.
+  chunk while accumulating the responses into an array. Returns a concatenated string of all
+  responses. Good for when you need to run the same query separately against each text
+  chunk.

 ```typescript
-import {
-  TextNode,
-  NodeWithScore,
-  ResponseSynthesizer,
-  CompactAndRefine
-} from "llamaindex";
+import { NodeWithScore, ResponseSynthesizer, TextNode } from "llamaindex";

 const responseSynthesizer = new ResponseSynthesizer();

@@ -42,7 +37,7 @@ const nodesWithScore: NodeWithScore[] = [

 const response = await responseSynthesizer.synthesize(
  "What age am I?",
-  nodesWithScore
+  nodesWithScore,
 );
 console.log(response.response);
 ```
@@ -4,10 +4,10 @@ sidebar_position: 5

 # Retriever

-A retriever in LlamaIndex is what is used to fetch `Node`s from an index using a query string. For example, a `ListIndexRetriever` will fetch all nodes no matter the query. Meanwhile, a `VectorIndexRetriever` will only fetch the top-k most similar nodes.
+A retriever in LlamaIndex is what is used to fetch `Node`s from an index using a query string. Aa `VectorIndexRetriever` will fetch the top-k most similar nodes. Meanwhile, a `SummaryIndexRetriever` will fetch all nodes no matter the query.

 ```typescript
-const retriever = vector_index.asRetriever()
+const retriever = vector_index.asRetriever();
 retriever.similarityTopK = 3;

 // Fetch nodes!
@@ -16,6 +16,6 @@ const nodesWithScore = await retriever.retrieve("query string");

 ## API Reference

- [ListIndexRetriever](../../api/classes/ListIndexRetriever.md)
- [ListIndexLLMRetriever](../../api/classes/ListIndexLLMRetriever.md)
+- [SummaryIndexRetriever](../../api/classes/SummaryIndexRetriever.md)
+- [SummaryIndexLLMRetriever](../../api/classes/SummaryIndexLLMRetriever.md)
 - [VectorIndexRetriever](../../api/classes/VectorIndexRetriever.md)
@@ -11,10 +11,14 @@ Right now, only saving and loading from disk is supported, with future integrati
 ```typescript
 import { Document, VectorStoreIndex, storageContextFromDefaults } from "./src";

-const storageContext = await storageContextFromDefaults({ persistDir: "./storage" });
+const storageContext = await storageContextFromDefaults({
+  persistDir: "./storage",
+});

 const document = new Document({ text: "Test Text" });
-const index = await VectorStoreIndex.fromDocuments([document], { storageContext });
+const index = await VectorStoreIndex.fromDocuments([document], {
+  storageContext,
+});
 ```

 ## API Reference
@@ -25,7 +25,7 @@ async function main() {
  // Load essay from abramov.txt in Node
  const essay = await fs.readFile(
    "node_modules/llamaindex/examples/abramov.txt",
-    "utf-8"
+    "utf-8",
  );

  // Create Document object with essay
@@ -37,7 +37,7 @@ async function main() {
  // Query the index
  const queryEngine = index.asQueryEngine();
  const response = await queryEngine.query(
-    "What did the author do in college?"
+    "What did the author do in college?",
  );

  // Output response
@@ -1,5 +1,5 @@
-import React from "react";
 import clsx from "clsx";
+import React from "react";
 import styles from "./styles.module.css";

 type FeatureItem = {
@@ -18,7 +18,7 @@
 }

 /* For readability concerns, you should choose a lighter palette in dark mode. */
-[data-theme='dark'] {
+[data-theme="dark"] {
  --ifm-color-primary: #25c2a0;
  --ifm-color-primary-dark: #21af90;
  --ifm-color-primary-darker: #1fa588;
@@ -1,5 +1,38 @@
 # simple

+## 0.0.25
+
+### Patch Changes
+
+- Updated dependencies [4a5591b]
+- Updated dependencies [4a5591b]
+- Updated dependencies [4a5591b]
+  - llamaindex@0.0.27
+
+## 0.0.24
+
+### Patch Changes
+
+- Updated dependencies [5bb55bc]
+  - llamaindex@0.0.26
+
+## 0.0.23
+
+### Patch Changes
+
+- Updated dependencies [e21eca2]
+- Updated dependencies [40a8f07]
+- Updated dependencies [40a8f07]
+  - llamaindex@0.0.25
+
+## 0.0.22
+
+### Patch Changes
+
+- Updated dependencies [e4af7b3]
+- Updated dependencies [259fe63]
+  - llamaindex@0.0.24
+
 ## 0.0.21

 ### Patch Changes
@@ -1,9 +1,10 @@
 # Simple Examples

 Due to packaging, you will need to run these commands to get started.
+
 ```bash
-pnpm --filter llamaindex build
 pnpm install
+pnpm --filter llamaindex build
 ```

 Then run the examples with `ts-node`, for example `npx ts-node vectorIndex.ts`
@@ -0,0 +1,32 @@
+import {
+  Document,
+  KeywordTableIndex,
+  KeywordTableRetrieverMode,
+} from "llamaindex";
+import essay from "./essay";
+
+async function main() {
+  const document = new Document({ text: essay, id_: "essay" });
+  const index = await KeywordTableIndex.fromDocuments([document]);
+
+  const allModes: KeywordTableRetrieverMode[] = [
+    KeywordTableRetrieverMode.DEFAULT,
+    KeywordTableRetrieverMode.SIMPLE,
+    KeywordTableRetrieverMode.RAKE,
+  ];
+  allModes.forEach(async (mode) => {
+    const queryEngine = index.asQueryEngine({
+      retriever: index.asRetriever({
+        mode,
+      }),
+    });
+    const response = await queryEngine.query(
+      "What did the author do growing up?",
+    );
+    console.log(response.toString());
+  });
+}
+
+main().catch((e: Error) => {
+  console.error(e, e.stack);
+});
@@ -1,9 +1,9 @@
 import {
  Document,
-  TextNode,
  NodeWithScore,
  ResponseSynthesizer,
  SimpleNodeParser,
+  TextNode,
 } from "llamaindex";

 (async () => {
@@ -29,7 +29,7 @@ import {

  const response = await responseSynthesizer.synthesize(
    "What age am I?",
-    nodesWithScore
+    nodesWithScore,
  );
  console.log(response.response);
 })();
@@ -0,0 +1,68 @@
+import { MongoClient } from "mongodb";
+import { VectorStoreIndex } from "../../packages/core/src/indices";
+import { Document } from "../../packages/core/src/Node";
+import { SimpleMongoReader } from "../../packages/core/src/readers/SimpleMongoReader";
+
+import { stdin as input, stdout as output } from "node:process";
+import readline from "node:readline/promises";
+
+async function main() {
+  //Dummy test code
+  const query: object = { _id: "waldo" };
+  const options: object = {};
+  const projections: object = { embedding: 0 };
+  const limit: number = Infinity;
+  const uri: string = process.env.MONGODB_URI ?? "fake_uri";
+  const client: MongoClient = new MongoClient(uri);
+
+  //Where the real code starts
+  const MR = new SimpleMongoReader(client);
+  const documents: Document[] = await MR.loadData(
+    "data",
+    "posts",
+    1,
+    {},
+    options,
+    projections,
+  );
+
+  //
+  //If you need to look at low-level details of
+  // a queryEngine (for example, needing to check each individual node)
+  //
+
+  // Split text and create embeddings. Store them in a VectorStoreIndex
+  // var storageContext = await storageContextFromDefaults({});
+  // var serviceContext = serviceContextFromDefaults({});
+  // const docStore = storageContext.docStore;
+
+  // for (const doc of documents) {
+  //   docStore.setDocumentHash(doc.id_, doc.hash);
+  // }
+  // const nodes = serviceContext.nodeParser.getNodesFromDocuments(documents);
+  // console.log(nodes);
+
+  //
+  //Making Vector Store from documents
+  //
+
+  const index = await VectorStoreIndex.fromDocuments(documents);
+  // Create query engine
+  const queryEngine = index.asQueryEngine();
+
+  const rl = readline.createInterface({ input, output });
+  while (true) {
+    const query = await rl.question("Query: ");
+
+    if (!query) {
+      break;
+    }
+
+    const response = await queryEngine.query(query);
+
+    // Output response
+    console.log(response.toString());
+  }
+}
+
+main();
@@ -0,0 +1,89 @@
+import { Client } from "@notionhq/client";
+import { program } from "commander";
+import { NotionReader, VectorStoreIndex } from "llamaindex";
+import { stdin as input, stdout as output } from "node:process";
+// readline/promises is still experimental so not in @types/node yet
+// @ts-ignore
+import readline from "node:readline/promises";
+
+program
+  .argument("[page]", "Notion page id (must be provided)")
+  .action(async (page, _options, command) => {
+    // Initializing a client
+
+    if (!process.env.NOTION_TOKEN) {
+      console.log(
+        "No NOTION_TOKEN found in environment variables. You will need to register an integration https://www.notion.com/my-integrations and put it in your NOTION_TOKEN environment variable.",
+      );
+      return;
+    }
+
+    const notion = new Client({
+      auth: process.env.NOTION_TOKEN,
+    });
+
+    if (!page) {
+      const response = await notion.search({
+        filter: {
+          value: "page",
+          property: "object",
+        },
+        sort: {
+          direction: "descending",
+          timestamp: "last_edited_time",
+        },
+      });
+
+      const { results } = response;
+
+      if (results.length === 0) {
+        console.log(
+          "No pages found. You will need to share it with your integration. (tap the three dots on the top right, find Add connections, and add your integration)",
+        );
+        return;
+      } else {
+        const pages = results
+          .map((result) => {
+            if (!("url" in result)) {
+              return null;
+            }
+
+            return {
+              id: result.id,
+              url: result.url,
+            };
+          })
+          .filter((page) => page !== null);
+        console.log("Found pages:");
+        console.table(pages);
+        console.log(`To run, run ts-node ${command.name()} [page id]`);
+        return;
+      }
+    }
+
+    const reader = new NotionReader({ client: notion });
+    const documents = await reader.loadData(page);
+    console.log(documents);
+
+    // Split text and create embeddings. Store them in a VectorStoreIndex
+    const index = await VectorStoreIndex.fromDocuments(documents);
+
+    // Create query engine
+    const queryEngine = index.asQueryEngine();
+
+    const rl = readline.createInterface({ input, output });
+    while (true) {
+      const query = await rl.question("Query: ");
+
+      if (!query) {
+        break;
+      }
+
+      const response = await queryEngine.query(query);
+
+      // Output response
+      console.log(response.toString());
+    }
+  });
+
+program.parse();
@@ -1,12 +1,14 @@
 {
-  "version": "0.0.21",
+  "version": "0.0.25",
  "private": true,
  "name": "simple",
  "dependencies": {
+    "@notionhq/client": "^2.2.12",
+    "commander": "^11.0.0",
    "llamaindex": "workspace:*"
  },
  "devDependencies": {
-    "@types/node": "^18.17.6"
+    "@types/node": "^18.17.12"
  },
  "scripts": {
    "lint": "eslint ."
@@ -0,0 +1,32 @@
+import {
+  Document,
+  KeywordTableIndex,
+  KeywordTableRetrieverMode,
+} from "llamaindex";
+import essay from "./essay";
+
+async function main() {
+  const document = new Document({ text: essay, id_: "essay" });
+  const index = await KeywordTableIndex.fromDocuments([document]);
+
+  const allModes: KeywordTableRetrieverMode[] = [
+    KeywordTableRetrieverMode.DEFAULT,
+    KeywordTableRetrieverMode.SIMPLE,
+    KeywordTableRetrieverMode.RAKE,
+  ];
+  allModes.forEach(async (mode) => {
+    const queryEngine = index.asQueryEngine({
+      retriever: index.asRetriever({
+        mode,
+      }),
+    });
+    const response = await queryEngine.query(
+      "What did the author do growing up?",
+    );
+    console.log(response.toString());
+  });
+}
+
+main().catch((e: Error) => {
+  console.error(e, e.stack);
+});
@@ -1,9 +1,9 @@
 import {
  Document,
-  TextNode,
  NodeWithScore,
  ResponseSynthesizer,
  SimpleNodeParser,
+  TextNode,
 } from "llamaindex";

 (async () => {
@@ -29,7 +29,7 @@ import {

  const response = await responseSynthesizer.synthesize(
    "What age am I?",
-    nodesWithScore
+    nodesWithScore,
  );
  console.log(response.response);
 })();
@@ -0,0 +1,89 @@
+import { Client } from "@notionhq/client";
+import { program } from "commander";
+import { NotionReader, VectorStoreIndex } from "llamaindex";
+import { stdin as input, stdout as output } from "node:process";
+// readline/promises is still experimental so not in @types/node yet
+// @ts-ignore
+import readline from "node:readline/promises";
+
+program
+  .argument("[page]", "Notion page id (must be provided)")
+  .action(async (page, _options, command) => {
+    // Initializing a client
+
+    if (!process.env.NOTION_TOKEN) {
+      console.log(
+        "No NOTION_TOKEN found in environment variables. You will need to register an integration https://www.notion.com/my-integrations and put it in your NOTION_TOKEN environment variable.",
+      );
+      return;
+    }
+
+    const notion = new Client({
+      auth: process.env.NOTION_TOKEN,
+    });
+
+    if (!page) {
+      const response = await notion.search({
+        filter: {
+          value: "page",
+          property: "object",
+        },
+        sort: {
+          direction: "descending",
+          timestamp: "last_edited_time",
+        },
+      });
+
+      const { results } = response;
+
+      if (results.length === 0) {
+        console.log(
+          "No pages found. You will need to share it with your integration. (tap the three dots on the top right, find Add connections, and add your integration)",
+        );
+        return;
+      } else {
+        const pages = results
+          .map((result) => {
+            if (!("url" in result)) {
+              return null;
+            }
+
+            return {
+              id: result.id,
+              url: result.url,
+            };
+          })
+          .filter((page) => page !== null);
+        console.log("Found pages:");
+        console.table(pages);
+        console.log(`To run, run ts-node ${command.name()} [page id]`);
+        return;
+      }
+    }
+
+    const reader = new NotionReader({ client: notion });
+    const documents = await reader.loadData(page);
+    console.log(documents);
+
+    // Split text and create embeddings. Store them in a VectorStoreIndex
+    const index = await VectorStoreIndex.fromDocuments(documents);
+
+    // Create query engine
+    const queryEngine = index.asQueryEngine();
+
+    const rl = readline.createInterface({ input, output });
+    while (true) {
+      const query = await rl.question("Query: ");
+
+      if (!query) {
+        break;
+      }
+
+      const response = await queryEngine.query(query);
+
+      // Output response
+      console.log(response.toString());
+    }
+  });
+
+program.parse();
@@ -16,8 +16,8 @@
    "eslint": "^7.32.0",
    "eslint-config-custom": "workspace:*",
    "husky": "^8.0.3",
-    "jest": "^29.6.3",
-    "prettier": "^3.0.2",
+    "jest": "^29.6.4",
+    "prettier": "^3.0.3",
    "prettier-plugin-organize-imports": "^3.2.3",
    "ts-jest": "^29.1.1",
    "turbo": "^1.10.13"
@@ -1,5 +1,34 @@
 # llamaindex

+## 0.0.27
+
+### Patch Changes
+
+- 4a5591b: Chat History summarization (thanks @marcusschiesser)
+- 4a5591b: Notion database support (thanks @TomPenguin)
+- 4a5591b: KeywordIndex (thanks @swk777)
+
+## 0.0.26
+
+### Patch Changes
+
+- 5bb55bc: Add notion loader (thank you @TomPenguin!)
+
+## 0.0.25
+
+### Patch Changes
+
+- e21eca2: OpenAI 4.3.1 and Anthropic 0.6.2
+- 40a8f07: Update READMEs (thanks @andfk)
+- 40a8f07: Bug: missing exports from storage (thanks @aashutoshrathi)
+
+## 0.0.24
+
+### Patch Changes
+
+- e4af7b3: Renamed ListIndex to SummaryIndex to better indicate its use.
+- 259fe63: Strong types for prompts.
+
 ## 0.0.23

 ### Patch Changes
@@ -20,7 +20,7 @@ In a new folder:
 export OPENAI_API_KEY="sk-......" # Replace with your key from https://platform.openai.com/account/api-keys
 pnpm init
 pnpm install typescript
-pnpm exec tsc –-init # if needed
+pnpm exec tsc --init # if needed
 pnpm install llamaindex
 pnpm install @types/node
 ```
@@ -36,7 +36,7 @@ async function main() {
  // Load essay from abramov.txt in Node
  const essay = await fs.readFile(
    "node_modules/llamaindex/examples/abramov.txt",
-    "utf-8"
+    "utf-8",
  );

  // Create Document object with essay
@@ -48,7 +48,7 @@ async function main() {
  // Query the index
  const queryEngine = index.asQueryEngine();
  const response = await queryEngine.query(
-    "What did the author do in college?"
+    "What did the author do in college?",
  );

  // Output response
@@ -61,7 +61,7 @@ main();
 Then you can run it using

 ```bash
-pnpm dlx ts-node example.ts
+pnpx ts-node example.ts
 ```

 ## Playground
@@ -1,23 +1,28 @@
 {
  "name": "llamaindex",
-  "version": "0.0.23",
+  "version": "0.0.27",
  "dependencies": {
-    "@anthropic-ai/sdk": "^0.6.1",
+    "@anthropic-ai/sdk": "^0.6.2",
+    "@notionhq/client": "^2.2.13",
    "lodash": "^4.17.21",
-    "openai": "^4.2.0",
+    "md-utils-ts": "^2.0.0",
+    "mongodb": "^6.0.0",
+    "notion-md-crawler": "^0.0.2",
+    "openai": "^4.7.0",
    "papaparse": "^5.4.1",
    "pdf-parse": "^1.1.1",
-    "replicate": "^0.16.1",
+    "rake-modified": "^1.0.8",
+    "replicate": "^0.18.0",
    "tiktoken-node": "^0.0.6",
-    "uuid": "^9.0.0",
+    "uuid": "^9.0.1",
    "wink-nlp": "^1.14.3"
  },
  "devDependencies": {
-    "@types/lodash": "^4.14.197",
-    "@types/node": "^18.17.9",
+    "@types/lodash": "^4.14.198",
+    "@types/node": "^18.17.15",
    "@types/papaparse": "^5.3.8",
    "@types/pdf-parse": "^1.1.1",
-    "@types/uuid": "^9.0.2",
+    "@types/uuid": "^9.0.4",
    "node-stdlib-browser": "^1.2.0",
    "tsup": "^7.2.0"
  },
@@ -1,4 +1,7 @@
 import { v4 as uuidv4 } from "uuid";
+import { Event } from "./callbacks/CallbackManager";
+import { ChatHistory, SimpleChatHistory } from "./ChatHistory";
+import { ChatMessage, LLM, OpenAI } from "./llm/LLM";
 import { TextNode } from "./Node";
 import {
  CondenseQuestionPrompt,
@@ -11,8 +14,6 @@ import { BaseQueryEngine } from "./QueryEngine";
 import { Response } from "./Response";
 import { BaseRetriever } from "./Retriever";
 import { ServiceContext, serviceContextFromDefaults } from "./ServiceContext";
-import { Event } from "./callbacks/CallbackManager";
-import { ChatMessage, LLM, OpenAI } from "./llm/LLM";

 /**
 * A ChatEngine is used to handle back and forth chats between the application and the LLM.
@@ -188,3 +189,28 @@ export class ContextChatEngine implements ChatEngine {
    this.chatHistory = [];
  }
 }
+
+/**
+ * HistoryChatEngine is a ChatEngine that uses a ChatHistory to keep track of the chat history. This is an example with the same behavior as SimpleChatEngine
+ * TODO: generally use the ChatHistory instead of ChatMessage[] - breaking change
+ */
+export class HistoryChatEngine implements ChatEngine {
+  chatHistory: ChatHistory;
+  llm: LLM;
+
+  constructor(init?: Partial<HistoryChatEngine>) {
+    this.chatHistory = init?.chatHistory ?? new SimpleChatHistory();
+    this.llm = init?.llm ?? new OpenAI();
+  }
+
+  async chat(message: string): Promise<Response> {
+    this.chatHistory.addMessage({ content: message, role: "user" });
+    const response = await this.llm.chat(this.chatHistory.messages);
+    this.chatHistory.addMessage(response.message);
+    return new Response(response.message.content);
+  }
+
+  reset() {
+    this.chatHistory.reset();
+  }
+}
@@ -0,0 +1,73 @@
+import { ChatMessage, LLM, OpenAI } from "./llm/LLM";
+import {
+  defaultSummaryPrompt,
+  messagesToHistoryStr,
+  SummaryPrompt,
+} from "./Prompt";
+
+/**
+ * A ChatHistory is used to keep the state of back and forth chat messages
+ */
+export interface ChatHistory {
+  messages: ChatMessage[];
+  /**
+   * Adds a message to the chat history.
+   * @param message
+   */
+  addMessage(message: ChatMessage): Promise<void>;
+
+  /**
+   * Resets the chat history so that it's empty.
+   */
+  reset(): void;
+}
+
+export class SimpleChatHistory implements ChatHistory {
+  messages: ChatMessage[];
+
+  constructor(init?: Partial<SimpleChatHistory>) {
+    this.messages = init?.messages ?? [];
+  }
+
+  async addMessage(message: ChatMessage) {
+    this.messages.push(message);
+  }
+
+  reset() {
+    this.messages = [];
+  }
+}
+
+export class SummaryChatHistory implements ChatHistory {
+  messages: ChatMessage[];
+  summaryPrompt: SummaryPrompt;
+  llm: LLM;
+
+  constructor(init?: Partial<SummaryChatHistory>) {
+    this.messages = init?.messages ?? [];
+    this.summaryPrompt = init?.summaryPrompt ?? defaultSummaryPrompt;
+    this.llm = init?.llm ?? new OpenAI();
+  }
+
+  private async summarize() {
+    const chatHistoryStr = messagesToHistoryStr(this.messages);
+
+    const response = await this.llm.complete(
+      this.summaryPrompt({ context: chatHistoryStr }),
+    );
+
+    this.messages = [{ content: response.message.content, role: "system" }];
+  }
+
+  async addMessage(message: ChatMessage) {
+    // TODO: check if summarization is necessary
+    // TBD what are good conditions, e.g. depending on the context length of the LLM?
+    // for now we just have a dummy implementation at always summarizes the messages
+    await this.summarize();
+    this.messages.push(message);
+  }
+
+  reset() {
+    this.messages = [];
+  }
+}
@@ -8,7 +8,7 @@ import {
  getAzureModel,
  shouldUseAzure,
 } from "./llm/azure";
-import { getOpenAISession, OpenAISession } from "./llm/openai";
+import { OpenAISession, getOpenAISession } from "./llm/openai";
 import { VectorStoreQueryMode } from "./storage/vectorStore/types";

 /**
@@ -280,9 +280,6 @@ export class OpenAIEmbedding extends BaseEmbedding {
  }

  private async getOpenAIEmbedding(input: string) {
-    input = input.replace(/\n/g, " ");
-    //^ NOTE this performance helper is in the OpenAI python library but may not be in the JS library
-
    const { data } = await this.session.openai.embeddings.create({
      model: this.model,
      input,
@@ -1,5 +1,5 @@
-import { Event, EventTag, EventType } from "./callbacks/CallbackManager";
 import { v4 as uuidv4 } from "uuid";
+import { Event, EventTag, EventType } from "./callbacks/CallbackManager";

 /**
 * Helper class singleton
@@ -23,19 +23,23 @@ export enum MetadataMode {
  NONE = "NONE",
 }

-export interface RelatedNodeInfo {
+export type Metadata = Record<string, any>;
+
+export interface RelatedNodeInfo<T extends Metadata = Metadata> {
  nodeId: string;
  nodeType?: ObjectType;
-  metadata: Record<string, any>;
+  metadata: T;
  hash?: string;
 }

-export type RelatedNodeType = RelatedNodeInfo | RelatedNodeInfo[];
+export type RelatedNodeType<T extends Metadata = Metadata> =
+  | RelatedNodeInfo<T>
+  | RelatedNodeInfo<T>[];

 /**
 * Generic abstract class for retrievable nodes
 */
-export abstract class BaseNode {
+export abstract class BaseNode<T extends Metadata = Metadata> {
  /**
   * The unique ID of the Node/Document. The trailing underscore is here
   * to avoid collisions with the id keyword in Python.
@@ -46,13 +50,13 @@ export abstract class BaseNode {
  embedding?: number[];

  // Metadata fields
-  metadata: Record<string, any> = {};
+  metadata: T = {} as T;
  excludedEmbedMetadataKeys: string[] = [];
  excludedLlmMetadataKeys: string[] = [];
-  relationships: Partial<Record<NodeRelationship, RelatedNodeType>> = {};
+  relationships: Partial<Record<NodeRelationship, RelatedNodeType<T>>> = {};
  hash: string = "";

-  constructor(init?: Partial<BaseNode>) {
+  constructor(init?: Partial<BaseNode<T>>) {
    Object.assign(this, init);
  }

@@ -62,7 +66,7 @@ export abstract class BaseNode {
  abstract getMetadataStr(metadataMode: MetadataMode): string;
  abstract setContent(value: any): void;

-  get sourceNode(): RelatedNodeInfo | undefined {
+  get sourceNode(): RelatedNodeInfo<T> | undefined {
    const relationship = this.relationships[NodeRelationship.SOURCE];

    if (Array.isArray(relationship)) {
@@ -72,7 +76,7 @@ export abstract class BaseNode {
    return relationship;
  }

-  get prevNode(): RelatedNodeInfo | undefined {
+  get prevNode(): RelatedNodeInfo<T> | undefined {
    const relationship = this.relationships[NodeRelationship.PREVIOUS];

    if (Array.isArray(relationship)) {
@@ -84,7 +88,7 @@ export abstract class BaseNode {
    return relationship;
  }

-  get nextNode(): RelatedNodeInfo | undefined {
+  get nextNode(): RelatedNodeInfo<T> | undefined {
    const relationship = this.relationships[NodeRelationship.NEXT];

    if (Array.isArray(relationship)) {
@@ -94,7 +98,7 @@ export abstract class BaseNode {
    return relationship;
  }

-  get parentNode(): RelatedNodeInfo | undefined {
+  get parentNode(): RelatedNodeInfo<T> | undefined {
    const relationship = this.relationships[NodeRelationship.PARENT];

    if (Array.isArray(relationship)) {
@@ -104,7 +108,7 @@ export abstract class BaseNode {
    return relationship;
  }

-  get childNodes(): RelatedNodeInfo[] | undefined {
+  get childNodes(): RelatedNodeInfo<T>[] | undefined {
    const relationship = this.relationships[NodeRelationship.CHILD];

    if (!Array.isArray(relationship)) {
@@ -126,7 +130,7 @@ export abstract class BaseNode {
    return this.embedding;
  }

-  asRelatedNodeInfo(): RelatedNodeInfo {
+  asRelatedNodeInfo(): RelatedNodeInfo<T> {
    return {
      nodeId: this.id_,
      metadata: this.metadata,
@@ -146,7 +150,7 @@ export abstract class BaseNode {
 /**
 * TextNode is the default node type for text. Most common node type in LlamaIndex.TS
 */
-export class TextNode extends BaseNode {
+export class TextNode<T extends Metadata = Metadata> extends BaseNode<T> {
  text: string = "";
  startCharIdx?: number;
  endCharIdx?: number;
@@ -154,7 +158,7 @@ export class TextNode extends BaseNode {
  // metadataTemplate: NOTE write your own formatter if needed
  metadataSeparator: string = "\n";

-  constructor(init?: Partial<TextNode>) {
+  constructor(init?: Partial<TextNode<T>>) {
    super(init);
    Object.assign(this, init);

@@ -233,10 +237,10 @@ export class TextNode extends BaseNode {
 //   }
 // }

-export class IndexNode extends TextNode {
+export class IndexNode<T extends Metadata = Metadata> extends TextNode<T> {
  indexId: string = "";

-  constructor(init?: Partial<IndexNode>) {
+  constructor(init?: Partial<IndexNode<T>>) {
    super(init);
    Object.assign(this, init);

@@ -253,8 +257,8 @@ export class IndexNode extends TextNode {
 /**
 * A document is just a special text node with a docId.
 */
-export class Document extends TextNode {
-  constructor(init?: Partial<Document>) {
+export class Document<T extends Metadata = Metadata> extends TextNode<T> {
+  constructor(init?: Partial<Document<T>>) {
    super(init);
    Object.assign(this, init);

@@ -292,7 +296,7 @@ export function jsonToNode(json: any) {
 /**
 * A node with a similarity score
 */
-export interface NodeWithScore {
-  node: BaseNode;
-  score: number;
+export interface NodeWithScore<T extends Metadata = Metadata> {
+  node: BaseNode<T>;
+  score?: number;
 }
@@ -28,7 +28,7 @@ class OutputParserError extends Error {

  constructor(
    message: string,
-    options: { cause?: Error; output?: string } = {}
+    options: { cause?: Error; output?: string } = {},
  ) {
    // @ts-ignore
    super(message, options); // https://github.com/tc39/proposal-error-cause
@@ -53,30 +53,31 @@ class OutputParserError extends Error {
 * @param text A markdown block with JSON
 * @returns parsed JSON object
 */
-function parseJsonMarkdown(text: string) {
+export function parseJsonMarkdown(text: string) {
  text = text.trim();

-  const beginDelimiter = "```json";
-  const endDelimiter = "```";
+  const left_square = text.indexOf("[");
+  const left_brace = text.indexOf("{");

-  const beginIndex = text.indexOf(beginDelimiter);
-  const endIndex = text.indexOf(
-    endDelimiter,
-    beginIndex + beginDelimiter.length
-  );
-  if (beginIndex === -1 || endIndex === -1) {
-    throw new OutputParserError("Not a json markdown", { output: text });
+  var left: number;
+  var right: number;
+  if (left_square < left_brace && left_square != -1) {
+    left = left_square;
+    right = text.lastIndexOf("]");
+  } else {
+    left = left_brace;
+    right = text.lastIndexOf("}");
  }
-
-  const jsonText = text.substring(beginIndex + beginDelimiter.length, endIndex);
-
+  const jsonText = text.substring(left, right + 1);
  try {
+    //Single JSON object case
+    if (left_square === -1) {
+      return [JSON.parse(jsonText)];
+    }
+    //Multiple JSON object case.
    return JSON.parse(jsonText);
  } catch (e) {
-    throw new OutputParserError("Not a valid json", {
-      cause: e as Error,
-      output: text,
-    });
+    throw new OutputParserError("Not a json markdown", { output: text });
  }
 }

@@ -7,7 +7,9 @@ import { ToolMetadata } from "./Tool";
 * NOTE this is a different interface compared to LlamaIndex Python
 * NOTE 2: we default to empty string to make it easy to calculate prompt sizes
 */
-export type SimplePrompt = (input: Record<string, string>) => string;
+export type SimplePrompt = (
+  input: Record<string, string | undefined>,
+) => string;

 /*
 DEFAULT_TEXT_QA_PROMPT_TMPL = (
@@ -354,3 +356,34 @@ ${context}
 };

 export type ContextSystemPrompt = typeof defaultContextSystemPrompt;
+
+export const defaultKeywordExtractPrompt = ({
+  context = "",
+  maxKeywords = 10,
+}) => {
+  return `
+Some text is provided below. Given the text, extract up to ${maxKeywords} keywords from the text. Avoid stopwords.
+---------------------
+${context}
+---------------------
+Provide keywords in the following comma-separated format: 'KEYWORDS: <keywords>'
+`;
+};
+
+export type KeywordExtractPrompt = typeof defaultKeywordExtractPrompt;
+
+export const defaultQueryKeywordExtractPrompt = ({
+  question = "",
+  maxKeywords = 10,
+}) => {
+  return `(
+  "A question is provided below. Given the question, extract up to ${maxKeywords} "
+  "keywords from the text. Focus on extracting the keywords that we can use "
+  "to best lookup answers to the question. Avoid stopwords."
+  "---------------------"
+  "${question}"
+  "---------------------"
+  "Provide keywords in the following comma-separated format: 'KEYWORDS: <keywords>'"
+)`;
+};
+export type QueryKeywordExtractPrompt = typeof defaultQueryKeywordExtractPrompt;
@@ -1,3 +1,4 @@
+import { v4 as uuidv4 } from "uuid";
 import { NodeWithScore, TextNode } from "./Node";
 import {
  BaseQuestionGenerator,
@@ -7,10 +8,9 @@ import {
 import { Response } from "./Response";
 import { CompactAndRefine, ResponseSynthesizer } from "./ResponseSynthesizer";
 import { BaseRetriever } from "./Retriever";
-import { v4 as uuidv4 } from "uuid";
-import { Event } from "./callbacks/CallbackManager";
 import { ServiceContext, serviceContextFromDefaults } from "./ServiceContext";
 import { QueryEngineTool, ToolMetadata } from "./Tool";
+import { Event } from "./callbacks/CallbackManager";

 /**
 * A query engine is a question answerer that can use one or more steps.
@@ -33,7 +33,7 @@ export class RetrieverQueryEngine implements BaseQueryEngine {

  constructor(
    retriever: BaseRetriever,
-    responseSynthesizer?: ResponseSynthesizer
+    responseSynthesizer?: ResponseSynthesizer,
  ) {
    this.retriever = retriever;
    const serviceContext: ServiceContext | undefined =
@@ -122,7 +122,7 @@ export class SubQuestionQueryEngine implements BaseQueryEngine {
    };

    const subQNodes = await Promise.all(
-      subQuestions.map((subQ) => this.querySubQ(subQ, subQueryParentEvent))
+      subQuestions.map((subQ) => this.querySubQ(subQ, subQueryParentEvent)),
    );

    const nodes = subQNodes
@@ -133,7 +133,7 @@ export class SubQuestionQueryEngine implements BaseQueryEngine {

  private async querySubQ(
    subQ: SubQuestion,
-    parentEvent?: Event
+    parentEvent?: Event,
  ): Promise<NodeWithScore | null> {
    try {
      const question = subQ.subQuestion;
@@ -1,8 +1,8 @@
 import { BaseEmbedding, OpenAIEmbedding } from "./Embedding";
-import { LLM, OpenAI } from "./llm/LLM";
 import { NodeParser, SimpleNodeParser } from "./NodeParser";
 import { PromptHelper } from "./PromptHelper";
 import { CallbackManager } from "./callbacks/CallbackManager";
+import { LLM, OpenAI } from "./llm/LLM";

 /**
 * The ServiceContext is a collection of components that are used in different parts of the application.
@@ -47,7 +47,7 @@ export function serviceContextFromDefaults(options?: ServiceContextOptions) {

 export function serviceContextFromServiceContext(
  serviceContext: ServiceContext,
-  options: ServiceContextOptions
+  options: ServiceContextOptions,
 ) {
  const newServiceContext = { ...serviceContext };
  if (options.llm) {
@@ -1,29 +1,30 @@
 export * from "./ChatEngine";
-export * from "./constants";
 export * from "./Embedding";
 export * from "./GlobalsHelper";
-export * from "./llm/LLM";
 export * from "./Node";
 export * from "./NodeParser";
 export * from "./OutputParser";
 export * from "./Prompt";
-export * from "./QuestionGenerator";
 export * from "./QueryEngine";
+export * from "./QuestionGenerator";
 export * from "./Response";
 export * from "./ResponseSynthesizer";
 export * from "./Retriever";
 export * from "./ServiceContext";
 export * from "./TextSplitter";
 export * from "./Tool";
+export * from "./constants";
+export * from "./llm/LLM";

 export * from "./indices";

 export * from "./callbacks/CallbackManager";

-export * from "./readers/base";
-export * from "./readers/PDFReader";
 export * from "./readers/CSVReader";
 export * from "./readers/MarkdownReader";
+export * from "./readers/NotionReader";
+export * from "./readers/PDFReader";
 export * from "./readers/SimpleDirectoryReader";
+export * from "./readers/base";

 export * from "./storage";
@@ -4,9 +4,9 @@ import { BaseQueryEngine } from "../QueryEngine";
 import { ResponseSynthesizer } from "../ResponseSynthesizer";
 import { BaseRetriever } from "../Retriever";
 import { ServiceContext } from "../ServiceContext";
+import { StorageContext } from "../storage/StorageContext";
 import { BaseDocumentStore } from "../storage/docStore/types";
 import { BaseIndexStore } from "../storage/indexStore/types";
-import { StorageContext } from "../storage/StorageContext";
 import { VectorStore } from "../storage/vectorStore/types";

 /**
@@ -39,6 +39,7 @@ export abstract class IndexStruct {
 export enum IndexStructType {
  SIMPLE_DICT = "simple_dict",
  LIST = "list",
+  KEYWORD_TABLE = "keyword_table",
 }

 export class IndexDict extends IndexStruct {
@@ -106,6 +107,36 @@ export class IndexList extends IndexStruct {
  }
 }

+// A table of keywords mapping keywords to text chunks.
+export class KeywordTable extends IndexStruct {
+  table: Map<string, Set<string>> = new Map();
+  type: IndexStructType = IndexStructType.KEYWORD_TABLE;
+  addNode(keywords: string[], nodeId: string): void {
+    keywords.forEach((keyword) => {
+      if (!this.table.has(keyword)) {
+        this.table.set(keyword, new Set());
+      }
+      this.table.get(keyword)!.add(nodeId);
+    });
+  }
+
+  deleteNode(keywords: string[], nodeId: string) {
+    keywords.forEach((keyword) => {
+      if (this.table.has(keyword)) {
+        this.table.get(keyword)!.delete(nodeId);
+      }
+    });
+  }
+
+  toJson(): Record<string, unknown> {
+    return {
+      ...super.toJson(),
+      table: this.table,
+      type: this.type,
+    };
+  }
+}
+
 export interface BaseIndexInit<T> {
  serviceContext: ServiceContext;
  storageContext: StorageContext;
@@ -1,3 +1,4 @@
 export * from "./BaseIndex";
+export * from "./keyword";
 export * from "./summary";
 export * from "./vectorStore";
@@ -0,0 +1,269 @@
+import { BaseNode, Document, MetadataMode } from "../../Node";
+import { defaultKeywordExtractPrompt } from "../../Prompt";
+import { BaseQueryEngine, RetrieverQueryEngine } from "../../QueryEngine";
+import { ResponseSynthesizer } from "../../ResponseSynthesizer";
+import { BaseRetriever } from "../../Retriever";
+import {
+  ServiceContext,
+  serviceContextFromDefaults,
+} from "../../ServiceContext";
+import { StorageContext, storageContextFromDefaults } from "../../storage";
+import { BaseDocumentStore } from "../../storage/docStore/types";
+import {
+  BaseIndex,
+  BaseIndexInit,
+  IndexStructType,
+  KeywordTable,
+} from "../BaseIndex";
+import {
+  KeywordTableLLMRetriever,
+  KeywordTableRAKERetriever,
+  KeywordTableSimpleRetriever,
+} from "./KeywordTableIndexRetriever";
+import { extractKeywordsGivenResponse } from "./utils";
+
+export interface KeywordIndexOptions {
+  nodes?: BaseNode[];
+  indexStruct?: KeywordTable;
+  indexId?: string;
+  serviceContext?: ServiceContext;
+  storageContext?: StorageContext;
+}
+export enum KeywordTableRetrieverMode {
+  DEFAULT = "DEFAULT",
+  SIMPLE = "SIMPLE",
+  RAKE = "RAKE",
+}
+
+const KeywordTableRetrieverMap = {
+  [KeywordTableRetrieverMode.DEFAULT]: KeywordTableLLMRetriever,
+  [KeywordTableRetrieverMode.SIMPLE]: KeywordTableSimpleRetriever,
+  [KeywordTableRetrieverMode.RAKE]: KeywordTableRAKERetriever,
+};
+
+/**
+ * The KeywordTableIndex, an index that extracts keywords from each Node and builds a mapping from each keyword to the corresponding Nodes of that keyword.
+ */
+export class KeywordTableIndex extends BaseIndex<KeywordTable> {
+  constructor(init: BaseIndexInit<KeywordTable>) {
+    super(init);
+  }
+
+  static async init(options: KeywordIndexOptions): Promise<KeywordTableIndex> {
+    const storageContext =
+      options.storageContext ?? (await storageContextFromDefaults({}));
+    const serviceContext =
+      options.serviceContext ?? serviceContextFromDefaults({});
+    const { docStore, indexStore } = storageContext;
+
+    // Setup IndexStruct from storage
+    let indexStructs = (await indexStore.getIndexStructs()) as KeywordTable[];
+    let indexStruct: KeywordTable | null;
+
+    if (options.indexStruct && indexStructs.length > 0) {
+      throw new Error(
+        "Cannot initialize index with both indexStruct and indexStore",
+      );
+    }
+
+    if (options.indexStruct) {
+      indexStruct = options.indexStruct;
+    } else if (indexStructs.length == 1) {
+      indexStruct = indexStructs[0];
+    } else if (indexStructs.length > 1 && options.indexId) {
+      indexStruct = (await indexStore.getIndexStruct(
+        options.indexId,
+      )) as KeywordTable;
+    } else {
+      indexStruct = null;
+    }
+
+    // check indexStruct type
+    if (indexStruct && indexStruct.type !== IndexStructType.KEYWORD_TABLE) {
+      throw new Error(
+        "Attempting to initialize KeywordTableIndex with non-keyword table indexStruct",
+      );
+    }
+
+    if (indexStruct) {
+      if (options.nodes) {
+        throw new Error(
+          "Cannot initialize KeywordTableIndex with both nodes and indexStruct",
+        );
+      }
+    } else {
+      if (!options.nodes) {
+        throw new Error(
+          "Cannot initialize KeywordTableIndex without nodes or indexStruct",
+        );
+      }
+      indexStruct = await KeywordTableIndex.buildIndexFromNodes(
+        options.nodes,
+        storageContext.docStore,
+        serviceContext,
+      );
+
+      await indexStore.addIndexStruct(indexStruct);
+    }
+
+    return new KeywordTableIndex({
+      storageContext,
+      serviceContext,
+      docStore,
+      indexStore,
+      indexStruct,
+    });
+  }
+
+  asRetriever(options?: any): BaseRetriever {
+    const { mode = KeywordTableRetrieverMode.DEFAULT, ...otherOptions } =
+      options ?? {};
+    const KeywordTableRetriever =
+      KeywordTableRetrieverMap[mode as KeywordTableRetrieverMode];
+    if (KeywordTableRetriever) {
+      return new KeywordTableRetriever({ index: this, ...otherOptions });
+    }
+    throw new Error(`Unknown retriever mode: ${mode}`);
+  }
+
+  asQueryEngine(options?: {
+    retriever?: BaseRetriever;
+    responseSynthesizer?: ResponseSynthesizer;
+  }): BaseQueryEngine {
+    const { retriever, responseSynthesizer } = options ?? {};
+    return new RetrieverQueryEngine(
+      retriever ?? this.asRetriever(),
+      responseSynthesizer,
+    );
+  }
+
+  static async extractKeywords(
+    text: string,
+    serviceContext: ServiceContext,
+  ): Promise<Set<string>> {
+    const response = await serviceContext.llm.complete(
+      defaultKeywordExtractPrompt({
+        context: text,
+      }),
+    );
+    return extractKeywordsGivenResponse(response.message.content, "KEYWORDS:");
+  }
+
+  /**
+   * High level API: split documents, get keywords, and build index.
+   * @param documents
+   * @param storageContext
+   * @param serviceContext
+   * @returns
+   */
+  static async fromDocuments(
+    documents: Document[],
+    args: {
+      storageContext?: StorageContext;
+      serviceContext?: ServiceContext;
+    } = {},
+  ): Promise<KeywordTableIndex> {
+    let { storageContext, serviceContext } = args;
+    storageContext = storageContext ?? (await storageContextFromDefaults({}));
+    serviceContext = serviceContext ?? serviceContextFromDefaults({});
+    const docStore = storageContext.docStore;
+
+    docStore.addDocuments(documents, true);
+    for (const doc of documents) {
+      docStore.setDocumentHash(doc.id_, doc.hash);
+    }
+
+    const nodes = serviceContext.nodeParser.getNodesFromDocuments(documents);
+    const index = await KeywordTableIndex.init({
+      nodes,
+      storageContext,
+      serviceContext,
+    });
+    return index;
+  }
+
+  /**
+   * Get keywords for nodes and place them into the index.
+   * @param nodes
+   * @param serviceContext
+   * @param vectorStore
+   * @returns
+   */
+  static async buildIndexFromNodes(
+    nodes: BaseNode[],
+    docStore: BaseDocumentStore,
+    serviceContext: ServiceContext,
+  ): Promise<KeywordTable> {
+    const indexStruct = new KeywordTable();
+    await docStore.addDocuments(nodes, true);
+    for (const node of nodes) {
+      const keywords = await KeywordTableIndex.extractKeywords(
+        node.getContent(MetadataMode.LLM),
+        serviceContext,
+      );
+      indexStruct.addNode([...keywords], node.id_);
+    }
+    return indexStruct;
+  }
+
+  async insertNodes(nodes: BaseNode[]) {
+    for (let node of nodes) {
+      const keywords = await KeywordTableIndex.extractKeywords(
+        node.getContent(MetadataMode.LLM),
+        this.serviceContext,
+      );
+      this.indexStruct.addNode([...keywords], node.id_);
+    }
+  }
+
+  deleteNode(nodeId: string): void {
+    const keywordsToDelete: Set<string> = new Set();
+    for (const [keyword, existingNodeIds] of Object.entries(
+      this.indexStruct.table,
+    )) {
+      const index = existingNodeIds.indexOf(nodeId);
+      if (index !== -1) {
+        existingNodeIds.splice(index, 1);
+
+        // Delete keywords that have zero nodes
+        if (existingNodeIds.length === 0) {
+          keywordsToDelete.add(keyword);
+        }
+      }
+    }
+    this.indexStruct.deleteNode([...keywordsToDelete], nodeId);
+  }
+
+  async deleteNodes(nodeIds: string[], deleteFromDocStore: boolean) {
+    nodeIds.forEach((nodeId) => {
+      this.deleteNode(nodeId);
+    });
+
+    if (deleteFromDocStore) {
+      for (const nodeId of nodeIds) {
+        await this.docStore.deleteDocument(nodeId, false);
+      }
+    }
+
+    await this.storageContext.indexStore.addIndexStruct(this.indexStruct);
+  }
+
+  async deleteRefDoc(
+    refDocId: string,
+    deleteFromDocStore?: boolean,
+  ): Promise<void> {
+    const refDocInfo = await this.docStore.getRefDocInfo(refDocId);
+
+    if (!refDocInfo) {
+      return;
+    }
+
+    await this.deleteNodes(refDocInfo.nodeIds, false);
+
+    if (deleteFromDocStore) {
+      await this.docStore.deleteRefDoc(refDocId, false);
+    }
+
+    return;
+  }
+}
@@ -0,0 +1,119 @@
+import { NodeWithScore } from "../../Node";
+import {
+  defaultKeywordExtractPrompt,
+  defaultQueryKeywordExtractPrompt,
+  KeywordExtractPrompt,
+  QueryKeywordExtractPrompt,
+} from "../../Prompt";
+import { BaseRetriever } from "../../Retriever";
+import { ServiceContext } from "../../ServiceContext";
+import { BaseDocumentStore } from "../../storage/docStore/types";
+import { KeywordTable } from "../BaseIndex";
+import { KeywordTableIndex } from "./KeywordTableIndex";
+import {
+  extractKeywordsGivenResponse,
+  rakeExtractKeywords,
+  simpleExtractKeywords,
+} from "./utils";
+
+// Base Keyword Table Retriever
+abstract class BaseKeywordTableRetriever implements BaseRetriever {
+  protected index: KeywordTableIndex;
+  protected indexStruct: KeywordTable;
+  protected docstore: BaseDocumentStore;
+  protected serviceContext: ServiceContext;
+
+  protected maxKeywordsPerQuery: number; // Maximum number of keywords to extract from query.
+  protected numChunksPerQuery: number; // Maximum number of text chunks to query.
+  protected keywordExtractTemplate: KeywordExtractPrompt; // A Keyword Extraction Prompt
+  protected queryKeywordExtractTemplate: QueryKeywordExtractPrompt; // A Query Keyword Extraction Prompt
+
+  constructor({
+    index,
+    keywordExtractTemplate,
+    queryKeywordExtractTemplate,
+    maxKeywordsPerQuery = 10,
+    numChunksPerQuery = 10,
+  }: {
+    index: KeywordTableIndex;
+    keywordExtractTemplate?: KeywordExtractPrompt;
+    queryKeywordExtractTemplate?: QueryKeywordExtractPrompt;
+    maxKeywordsPerQuery: number;
+    numChunksPerQuery: number;
+  }) {
+    this.index = index;
+    this.indexStruct = index.indexStruct;
+    this.docstore = index.docStore;
+    this.serviceContext = index.serviceContext;
+
+    this.maxKeywordsPerQuery = maxKeywordsPerQuery;
+    this.numChunksPerQuery = numChunksPerQuery;
+    this.keywordExtractTemplate =
+      keywordExtractTemplate || defaultKeywordExtractPrompt;
+    this.queryKeywordExtractTemplate =
+      queryKeywordExtractTemplate || defaultQueryKeywordExtractPrompt;
+  }
+
+  abstract getKeywords(query: string): Promise<string[]>;
+
+  async retrieve(query: string): Promise<NodeWithScore[]> {
+    const keywords = await this.getKeywords(query);
+    const chunkIndicesCount: { [key: string]: number } = {};
+    const filteredKeywords = keywords.filter((keyword) =>
+      this.indexStruct.table.has(keyword),
+    );
+
+    for (let keyword of filteredKeywords) {
+      for (let nodeId of this.indexStruct.table.get(keyword) || []) {
+        chunkIndicesCount[nodeId] = (chunkIndicesCount[nodeId] ?? 0) + 1;
+      }
+    }
+
+    const sortedChunkIndices = Object.keys(chunkIndicesCount)
+      .sort((a, b) => chunkIndicesCount[b] - chunkIndicesCount[a])
+      .slice(0, this.numChunksPerQuery);
+
+    const sortedNodes = await this.docstore.getNodes(sortedChunkIndices);
+
+    return sortedNodes.map((node) => ({ node }));
+  }
+
+  getServiceContext(): ServiceContext {
+    return this.index.serviceContext;
+  }
+}
+
+// Extracts keywords using LLMs.
+export class KeywordTableLLMRetriever extends BaseKeywordTableRetriever {
+  async getKeywords(query: string): Promise<string[]> {
+    const response = await this.serviceContext.llm.complete(
+      this.queryKeywordExtractTemplate({
+        question: query,
+        maxKeywords: this.maxKeywordsPerQuery,
+      }),
+    );
+    const keywords = extractKeywordsGivenResponse(
+      response.message.content,
+      "KEYWORDS:",
+    );
+    return [...keywords];
+  }
+}
+
+// Extracts keywords using simple regex-based keyword extractor.
+export class KeywordTableSimpleRetriever extends BaseKeywordTableRetriever {
+  getKeywords(query: string): Promise<string[]> {
+    return Promise.resolve([
+      ...simpleExtractKeywords(query, this.maxKeywordsPerQuery),
+    ]);
+  }
+}
+
+// Extracts keywords using RAKE keyword extractor
+export class KeywordTableRAKERetriever extends BaseKeywordTableRetriever {
+  getKeywords(query: string): Promise<string[]> {
+    return Promise.resolve([
+      ...rakeExtractKeywords(query, this.maxKeywordsPerQuery),
+    ]);
+  }
+}
@@ -0,0 +1,9 @@
+export {
+  KeywordTableIndex,
+  KeywordTableRetrieverMode,
+} from "./KeywordTableIndex";
+export {
+  KeywordTableLLMRetriever,
+  KeywordTableRAKERetriever,
+  KeywordTableSimpleRetriever,
+} from "./KeywordTableIndexRetriever";
@@ -0,0 +1,81 @@
+// @ts-ignore
+import rake from "rake-modified";
+
+// Get subtokens from a list of tokens., filtering for stopwords.
+export function expandTokensWithSubtokens(tokens: Set<string>): Set<string> {
+  const results: Set<string> = new Set();
+  const regex: RegExp = /\w+/g;
+
+  for (let token of tokens) {
+    results.add(token);
+    const subTokens: RegExpMatchArray | null = token.match(regex);
+    if (subTokens && subTokens.length > 1) {
+      for (let w of subTokens) {
+        results.add(w);
+      }
+    }
+  }
+  return results;
+}
+
+export function extractKeywordsGivenResponse(
+  response: string,
+  startToken: string = "",
+  lowercase: boolean = true,
+): Set<string> {
+  const results: string[] = [];
+  response = response.trim();
+
+  if (response.startsWith(startToken)) {
+    response = response.substring(startToken.length);
+  }
+
+  const keywords: string[] = response.split(",");
+  for (let k of keywords) {
+    let rk: string = k;
+    if (lowercase) {
+      rk = rk.toLowerCase();
+    }
+    results.push(rk.trim());
+  }
+
+  return expandTokensWithSubtokens(new Set(results));
+}
+
+export function simpleExtractKeywords(
+  textChunk: string,
+  maxKeywords?: number,
+): Set<string> {
+  const regex: RegExp = /\w+/g;
+  let tokens: string[] = [...textChunk.matchAll(regex)].map((token) =>
+    token[0].toLowerCase().trim(),
+  );
+
+  // Creating a frequency map
+  const valueCounts: { [key: string]: number } = {};
+  for (let token of tokens) {
+    valueCounts[token] = (valueCounts[token] || 0) + 1;
+  }
+
+  // Sorting tokens by frequency
+  const sortedTokens: string[] = Object.keys(valueCounts).sort(
+    (a, b) => valueCounts[b] - valueCounts[a],
+  );
+
+  const keywords: string[] = maxKeywords
+    ? sortedTokens.slice(0, maxKeywords)
+    : sortedTokens;
+
+  return new Set(keywords);
+}
+
+export function rakeExtractKeywords(
+  textChunk: string,
+  maxKeywords?: number,
+): Set<string> {
+  const keywords = Object.keys(rake(textChunk));
+  const limitedKeywords = maxKeywords
+    ? keywords.slice(0, maxKeywords)
+    : keywords;
+  return new Set(limitedKeywords);
+}
@@ -1,9 +1,9 @@
-import { BaseNode, MetadataMode } from "../../Node";
 import _ from "lodash";
+import { BaseNode, MetadataMode } from "../../Node";

 export type NodeFormatterFunction = (summaryNodes: BaseNode[]) => string;
 export const defaultFormatNodeBatchFn: NodeFormatterFunction = (
-  summaryNodes: BaseNode[]
+  summaryNodes: BaseNode[],
 ): string => {
  return summaryNodes
    .map((node, idx) => {
@@ -20,13 +20,13 @@ export type ChoiceSelectParseResult = { [docNumber: number]: number };
 export type ChoiceSelectParserFunction = (
  answer: string,
  numChoices: number,
-  raiseErr?: boolean
+  raiseErr?: boolean,
 ) => ChoiceSelectParseResult;

 export const defaultParseChoiceSelectAnswerFn: ChoiceSelectParserFunction = (
  answer: string,
  numChoices: number,
-  raiseErr: boolean = false
+  raiseErr: boolean = false,
 ): ChoiceSelectParseResult => {
  // split the line into the answer number and relevance score portions
  const lineTokens: string[][] = answer
@@ -36,7 +36,7 @@ export const defaultParseChoiceSelectAnswerFn: ChoiceSelectParserFunction = (
      if (lineTokens.length !== 2) {
        if (raiseErr) {
          throw new Error(
-            `Invalid answer line: ${line}. Answer line must be of the form: answer_num: <int>, answer_relevance: <float>`
+            `Invalid answer line: ${line}. Answer line must be of the form: answer_num: <int>, answer_relevance: <float>`,
          );
        } else {
          return null;
@@ -55,7 +55,7 @@ export const defaultParseChoiceSelectAnswerFn: ChoiceSelectParserFunction = (
        if (docNum < 1 || docNum > numChoices) {
          if (raiseErr) {
            throw new Error(
-              `Invalid answer number: ${docNum}. Answer number must be between 1 and ${numChoices}`
+              `Invalid answer number: ${docNum}. Answer number must be between 1 and ${numChoices}`,
            );
          }
        } else {
@@ -68,6 +68,6 @@ export const defaultParseChoiceSelectAnswerFn: ChoiceSelectParserFunction = (
      }
      return parseResult;
    },
-    {}
+    {},
  );
 };
@@ -1,6 +1,6 @@
-import { VectorStoreIndex } from "./VectorStoreIndex";
 import { globalsHelper } from "../../GlobalsHelper";
 import { NodeWithScore } from "../../Node";
+import { BaseRetriever } from "../../Retriever";
 import { ServiceContext } from "../../ServiceContext";
 import { Event } from "../../callbacks/CallbackManager";
 import { DEFAULT_SIMILARITY_TOP_K } from "../../constants";
@@ -8,7 +8,7 @@ import {
  VectorStoreQuery,
  VectorStoreQueryMode,
 } from "../../storage/vectorStore/types";
-import { BaseRetriever } from "../../Retriever";
+import { VectorStoreIndex } from "./VectorStoreIndex";

 /**
 * VectorIndexRetriever retrieves nodes from a VectorIndex.
@@ -1,2 +1,2 @@
-export { VectorStoreIndex } from "./VectorStoreIndex";
 export { VectorIndexRetriever } from "./VectorIndexRetriever";
+export { VectorStoreIndex } from "./VectorStoreIndex";
@@ -1,6 +1,6 @@
 import Anthropic, {
-  ClientOptions,
  AI_PROMPT,
+  ClientOptions,
  HUMAN_PROMPT,
 } from "@anthropic-ai/sdk";
 import _ from "lodash";
@@ -38,7 +38,7 @@ const DEFAULT_API_VERSION = "2023-05-15";
 //^ NOTE: this will change over time, if you want to pin it, use a specific version

 export function getAzureConfigFromEnv(
-  init?: Partial<AzureOpenAIConfig> & { model?: string }
+  init?: Partial<AzureOpenAIConfig> & { model?: string },
 ): AzureOpenAIConfig {
  return {
    apiKey:
@@ -71,7 +71,7 @@ export function getAzureBaseUrl(config: AzureOpenAIConfig): string {

 export function getAzureModel(openAIModel: string) {
  for (const [key, value] of Object.entries(
-    ALL_AZURE_OPENAI_EMBEDDING_MODELS
+    ALL_AZURE_OPENAI_EMBEDDING_MODELS,
  )) {
    if (value.openAIModel === openAIModel) {
      return key;
@@ -1,5 +1,5 @@
-import OpenAI, { ClientOptions } from "openai";
 import _ from "lodash";
+import OpenAI, { ClientOptions } from "openai";

 export class AzureOpenAI extends OpenAI {
  protected override authHeaders() {
@@ -42,7 +42,7 @@ let defaultOpenAISession: { session: OpenAISession; options: ClientOptions }[] =
 * @returns
 */
 export function getOpenAISession(
-  options: ClientOptions & { azure?: boolean } = {}
+  options: ClientOptions & { azure?: boolean } = {},
 ) {
  let session = defaultOpenAISession.find((session) => {
    return _.isEqual(session.options, options);
@@ -11,7 +11,7 @@ export class ReplicateSession {
      this.replicateKey = process.env.REPLICATE_API_TOKEN;
    } else {
      throw new Error(
-        "Set Replicate token in REPLICATE_API_TOKEN env variable"
+        "Set Replicate token in REPLICATE_API_TOKEN env variable",
      );
    }

@@ -1,7 +1,7 @@
-import { DEFAULT_FS, GenericFileSystem } from "../storage/FileSystem";
 import Papa, { ParseConfig } from "papaparse";
-import { BaseReader } from "./base";
 import { Document } from "../Node";
+import { DEFAULT_FS, GenericFileSystem } from "../storage/FileSystem";
+import { BaseReader } from "./base";

 /**
 * papaparse-based csv parser
@@ -24,7 +24,7 @@ export class PapaCSVReader implements BaseReader {
    concatRows: boolean = true,
    colJoiner: string = ", ",
    rowJoiner: string = "\n",
-    papaConfig?: ParseConfig
+    papaConfig?: ParseConfig,
  ) {
    this.concatRows = concatRows;
    this.colJoiner = colJoiner;
@@ -40,7 +40,7 @@ export class PapaCSVReader implements BaseReader {
   */
  async loadData(
    file: string,
-    fs: GenericFileSystem = DEFAULT_FS
+    fs: GenericFileSystem = DEFAULT_FS,
  ): Promise<Document[]> {
    const fileContent: string = await fs.readFile(file, "utf-8");
    const result = Papa.parse(fileContent, this.papaConfig);
@@ -0,0 +1,67 @@
+import { Client } from "@notionhq/client";
+import { crawler, Crawler, Pages, pageToString } from "notion-md-crawler";
+import { Document } from "../Node";
+import { BaseReader } from "./base";
+
+type OptionalSerializers = Parameters<Crawler>[number]["serializers"];
+
+/**
+ * Options for initializing the NotionReader class
+ * @typedef {Object} NotionReaderOptions
+ * @property {Client} client - The Notion Client object for API interactions
+ * @property {OptionalSerializers} [serializers] - Option to customize serialization. See [the url](https://github.com/TomPenguin/notion-md-crawler/tree/main) for details.
+ */
+type NotionReaderOptions = {
+  client: Client;
+  serializers?: OptionalSerializers;
+};
+
+/**
+ * Notion pages are retrieved recursively and converted to Document objects.
+ * Notion Database can also be loaded, and [the serialization method can be customized](https://github.com/TomPenguin/notion-md-crawler/tree/main).
+ *
+ * [Note] To use this reader, must be created the Notion integration must be created in advance
+ * Please refer to [this document](https://www.notion.so/help/create-integrations-with-the-notion-api) for details.
+ */
+export class NotionReader implements BaseReader {
+  private crawl: ReturnType<Crawler>;
+
+  /**
+   * Constructor for the NotionReader class
+   * @param {NotionReaderOptions} options - Configuration options for the reader
+   */
+  constructor({ client, serializers }: NotionReaderOptions) {
+    this.crawl = crawler({ client, serializers });
+  }
+
+  /**
+   * Converts Pages to an array of Document objects
+   * @param {Pages} pages - The Notion pages to convert (Return value of `loadPages`)
+   * @returns {Document[]} An array of Document objects
+   */
+  toDocuments(pages: Pages): Document[] {
+    return Object.values(pages).map((page) => {
+      const text = pageToString(page);
+      return new Document({ text, metadata: page.metadata });
+    });
+  }
+
+  /**
+   * Loads recursively the Notion page with the specified root page ID.
+   * @param {string} rootPageId - The root Notion page ID
+   * @returns {Promise<Pages>} A Promise that resolves to a Pages object(Convertible with the `toDocuments` method)
+   */
+  async loadPages(rootPageId: string): Promise<Pages> {
+    return this.crawl(rootPageId);
+  }
+
+  /**
+   * Loads recursively Notion pages and converts them to an array of Document objects
+   * @param {string} rootPageId - The root Notion page ID
+   * @returns {Promise<Document[]>} A Promise that resolves to an array of Document objects
+   */
+  async loadData(rootPageId: string): Promise<Document[]> {
+    const pages = await this.loadPages(rootPageId);
+    return this.toDocuments(pages);
+  }
+}
@@ -1,8 +1,8 @@
+import pdfParse from "pdf-parse";
 import { Document } from "../Node";
-import { BaseReader } from "./base";
 import { GenericFileSystem } from "../storage/FileSystem";
 import { DEFAULT_FS } from "../storage/constants";
-import pdfParse from "pdf-parse";
+import { BaseReader } from "./base";

 /**
 * Read the text of a PDF
@@ -10,7 +10,7 @@ import pdfParse from "pdf-parse";
 export class PDFReader implements BaseReader {
  async loadData(
    file: string,
-    fs: GenericFileSystem = DEFAULT_FS
+    fs: GenericFileSystem = DEFAULT_FS,
  ): Promise<Document[]> {
    const dataBuffer = (await fs.readFile(file)) as any;
    const data = await pdfParse(dataBuffer);
@@ -1,11 +1,11 @@
 import _ from "lodash";
 import { Document } from "../Node";
-import { BaseReader } from "./base";
 import { CompleteFileSystem, walk } from "../storage/FileSystem";
 import { DEFAULT_FS } from "../storage/constants";
-import { PDFReader } from "./PDFReader";
 import { PapaCSVReader } from "./CSVReader";
 import { MarkdownReader } from "./MarkdownReader";
+import { PDFReader } from "./PDFReader";
+import { BaseReader } from "./base";

 /**
 * Read a .txt file
@@ -13,7 +13,7 @@ import { MarkdownReader } from "./MarkdownReader";
 export class TextFileReader implements BaseReader {
  async loadData(
    file: string,
-    fs: CompleteFileSystem = DEFAULT_FS as CompleteFileSystem
+    fs: CompleteFileSystem = DEFAULT_FS as CompleteFileSystem,
  ): Promise<Document[]> {
    const dataBuffer = await fs.readFile(file, "utf-8");
    return [new Document({ text: dataBuffer, id_: file })];
@@ -0,0 +1,51 @@
+import { MongoClient } from "mongodb";
+import { Document } from "../Node";
+import { BaseReader } from "./base";
+
+/**
+ * Read in from MongoDB
+ */
+export class SimpleMongoReader implements BaseReader {
+  private client: MongoClient;
+
+  constructor(client: MongoClient) {
+    this.client = client;
+  }
+
+  /**
+   * Loads data from MongoDB collection
+   * @param {string} db_name - The name of the database to load.
+   * @param {string} collection_name - The name of the collection to load.
+   * @param {Number} [max_docs = 0] - Maximum number of documents to return. 0 means no limit.
+   * @param {Record<string, any>} [query_dict={}] - Specific query, as specified by MongoDB NodeJS documentation.
+   * @param {Record<string, any>} [query_options={}] - Specific query options, as specified by MongoDB NodeJS documentation.
+   * @param {Record<string, any>} [projection = {}] - Projection options, as specified by MongoDB NodeJS documentation.
+   * @returns {Promise<Document[]>}
+   */
+  async loadData(
+    db_name: string,
+    collection_name: string,
+    max_docs = 0,
+    //For later: Think about whether we want to pass generic objects in...
+    query_dict: Record<string, any> = {},
+    query_options: Record<string, any> = {},
+    projection: Record<string, any> = {},
+  ): Promise<Document[]> {
+    //Get items from collection using built-in functions
+    const cursor: Partial<Document>[] = await this.client
+      .db(db_name)
+      .collection(collection_name)
+      .find(query_dict, query_options)
+      .limit(max_docs)
+      .project(projection)
+      .toArray();
+
+    //Aggregate results and return
+    const documents: Document[] = [];
+    cursor.forEach((element: Partial<Document>) => {
+      //For later: Metadata filtering
+      documents.push(new Document({ text: JSON.stringify(element) }));
+    });
+    return documents;
+  }
+}
@@ -73,7 +73,7 @@ export const DEFAULT_FS: GenericFileSystem | CompleteFileSystem =
 */
 export async function exists(
  fs: GenericFileSystem,
-  path: string
+  path: string,
 ): Promise<boolean> {
  try {
    await fs.access(path);
@@ -90,11 +90,11 @@ export async function exists(
 */
 export async function* walk(
  fs: WalkableFileSystem,
-  dirPath: string
+  dirPath: string,
 ): AsyncIterable<string> {
  if (fs instanceof InMemoryFileSystem) {
    throw new Error(
-      "The InMemoryFileSystem does not support directory traversal."
+      "The InMemoryFileSystem does not support directory traversal.",
    );
  }

@@ -1,15 +1,11 @@
-import { BaseDocumentStore } from "./docStore/types";
-import { BaseIndexStore } from "./indexStore/types";
-import { VectorStore } from "./vectorStore/types";
-import { SimpleDocumentStore } from "./docStore/SimpleDocumentStore";
-import { SimpleIndexStore } from "./indexStore/SimpleIndexStore";
-import { SimpleVectorStore } from "./vectorStore/SimpleVectorStore";
 import { GenericFileSystem } from "./FileSystem";
-import {
-  DEFAULT_PERSIST_DIR,
-  DEFAULT_FS,
-  DEFAULT_NAMESPACE,
-} from "./constants";
+import { DEFAULT_FS, DEFAULT_NAMESPACE } from "./constants";
+import { SimpleDocumentStore } from "./docStore/SimpleDocumentStore";
+import { BaseDocumentStore } from "./docStore/types";
+import { SimpleIndexStore } from "./indexStore/SimpleIndexStore";
+import { BaseIndexStore } from "./indexStore/types";
+import { SimpleVectorStore } from "./vectorStore/SimpleVectorStore";
+import { VectorStore } from "./vectorStore/types";

 export interface StorageContext {
  docStore: BaseDocumentStore;
@@ -43,7 +39,7 @@ export async function storageContextFromDefaults({
      (await SimpleDocumentStore.fromPersistDir(
        persistDir,
        DEFAULT_NAMESPACE,
-        fs
+        fs,
      ));
    indexStore =
      indexStore || (await SimpleIndexStore.fromPersistDir(persistDir, fs));
@@ -1,15 +1,15 @@
-import * as path from "path";
 import _ from "lodash";
-import { KVDocumentStore } from "./KVDocumentStore";
-import { SimpleKVStore } from "../kvStore/SimpleKVStore";
-import { BaseInMemoryKVStore } from "../kvStore/types";
+import * as path from "path";
 import { GenericFileSystem } from "../FileSystem";
 import {
-  DEFAULT_PERSIST_DIR,
-  DEFAULT_NAMESPACE,
  DEFAULT_DOC_STORE_PERSIST_FILENAME,
  DEFAULT_FS,
+  DEFAULT_NAMESPACE,
+  DEFAULT_PERSIST_DIR,
 } from "../constants";
+import { SimpleKVStore } from "../kvStore/SimpleKVStore";
+import { BaseInMemoryKVStore } from "../kvStore/types";
+import { KVDocumentStore } from "./KVDocumentStore";

 type SaveDict = Record<string, any>;

@@ -26,23 +26,23 @@ export class SimpleDocumentStore extends KVDocumentStore {
  static async fromPersistDir(
    persistDir: string = DEFAULT_PERSIST_DIR,
    namespace?: string,
-    fsModule?: GenericFileSystem
+    fsModule?: GenericFileSystem,
  ): Promise<SimpleDocumentStore> {
    const persistPath = path.join(
      persistDir,
-      DEFAULT_DOC_STORE_PERSIST_FILENAME
+      DEFAULT_DOC_STORE_PERSIST_FILENAME,
    );
    return await SimpleDocumentStore.fromPersistPath(
      persistPath,
      namespace,
-      fsModule
+      fsModule,
    );
  }

  static async fromPersistPath(
    persistPath: string,
    namespace?: string,
-    fs?: GenericFileSystem
+    fs?: GenericFileSystem,
  ): Promise<SimpleDocumentStore> {
    fs = fs || DEFAULT_FS;
    const simpleKVStore = await SimpleKVStore.fromPersistPath(persistPath, fs);
@@ -52,9 +52,9 @@ export class SimpleDocumentStore extends KVDocumentStore {
  async persist(
    persistPath: string = path.join(
      DEFAULT_PERSIST_DIR,
-      DEFAULT_DOC_STORE_PERSIST_FILENAME
+      DEFAULT_DOC_STORE_PERSIST_FILENAME,
    ),
-    fs?: GenericFileSystem
+    fs?: GenericFileSystem,
  ): Promise<void> {
    fs = fs || DEFAULT_FS;
    if (
@@ -1,4 +1,4 @@
-import { BaseNode, Document, TextNode, ObjectType } from "../../Node";
+import { BaseNode, Document, ObjectType, TextNode } from "../../Node";

 const TYPE_KEY = "__type__";
 const DATA_KEY = "__data__";
@@ -1,7 +1,11 @@
-export * from "./constants";
 export * from "./FileSystem";
 export * from "./StorageContext";
-export * from "./vectorStore/types";
+export * from "./constants";
 export { SimpleDocumentStore } from "./docStore/SimpleDocumentStore";
+export * from "./docStore/types";
 export { SimpleIndexStore } from "./indexStore/SimpleIndexStore";
+export * from "./indexStore/types";
+export { SimpleKVStore } from "./kvStore/SimpleKVStore";
+export * from "./kvStore/types";
 export { SimpleVectorStore } from "./vectorStore/SimpleVectorStore";
+export * from "./vectorStore/types";
@@ -1,7 +1,7 @@
-import { BaseKVStore } from "../kvStore/types";
-import { IndexStruct, jsonToIndexStruct } from "../../indices/BaseIndex";
 import _ from "lodash";
+import { IndexStruct, jsonToIndexStruct } from "../../indices/BaseIndex";
 import { DEFAULT_NAMESPACE } from "../constants";
+import { BaseKVStore } from "../kvStore/types";
 import { BaseIndexStore } from "./types";

 export class KVIndexStore extends BaseIndexStore {
@@ -1,14 +1,13 @@
 import * as path from "path";
-import * as _ from "lodash";
-import { BaseInMemoryKVStore } from "../kvStore/types";
-import { SimpleKVStore, DataType } from "../kvStore/SimpleKVStore";
-import { KVIndexStore } from "./KVIndexStore";
-import {
-  DEFAULT_PERSIST_DIR,
-  DEFAULT_INDEX_STORE_PERSIST_FILENAME,
-  DEFAULT_FS,
-} from "../constants";
 import { GenericFileSystem } from "../FileSystem";
+import {
+  DEFAULT_FS,
+  DEFAULT_INDEX_STORE_PERSIST_FILENAME,
+  DEFAULT_PERSIST_DIR,
+} from "../constants";
+import { DataType, SimpleKVStore } from "../kvStore/SimpleKVStore";
+import { BaseInMemoryKVStore } from "../kvStore/types";
+import { KVIndexStore } from "./KVIndexStore";

 export class SimpleIndexStore extends KVIndexStore {
  private kvStore: BaseInMemoryKVStore;
@@ -21,18 +20,18 @@ export class SimpleIndexStore extends KVIndexStore {

  static async fromPersistDir(
    persistDir: string = DEFAULT_PERSIST_DIR,
-    fs: GenericFileSystem = DEFAULT_FS
+    fs: GenericFileSystem = DEFAULT_FS,
  ): Promise<SimpleIndexStore> {
    const persistPath = path.join(
      persistDir,
-      DEFAULT_INDEX_STORE_PERSIST_FILENAME
+      DEFAULT_INDEX_STORE_PERSIST_FILENAME,
    );
    return this.fromPersistPath(persistPath, fs);
  }

  static async fromPersistPath(
    persistPath: string,
-    fs: GenericFileSystem = DEFAULT_FS
+    fs: GenericFileSystem = DEFAULT_FS,
  ): Promise<SimpleIndexStore> {
    let simpleKVStore = await SimpleKVStore.fromPersistPath(persistPath, fs);
    return new SimpleIndexStore(simpleKVStore);
@@ -40,7 +39,7 @@ export class SimpleIndexStore extends KVIndexStore {

  async persist(
    persistPath: string = DEFAULT_PERSIST_DIR,
-    fs: GenericFileSystem = DEFAULT_FS
+    fs: GenericFileSystem = DEFAULT_FS,
  ): Promise<void> {
    await this.kvStore.persist(persistPath, fs);
  }
@@ -1,8 +1,8 @@
 import { IndexStruct } from "../../indices/BaseIndex";
 import { GenericFileSystem } from "../FileSystem";
 import {
-  DEFAULT_PERSIST_DIR,
  DEFAULT_INDEX_STORE_PERSIST_FILENAME,
+  DEFAULT_PERSIST_DIR,
 } from "../constants";

 const defaultPersistPath = `${DEFAULT_PERSIST_DIR}/${DEFAULT_INDEX_STORE_PERSIST_FILENAME}`;
@@ -18,7 +18,7 @@ export abstract class BaseIndexStore {

  async persist(
    persistPath: string = defaultPersistPath,
-    fs?: GenericFileSystem
+    fs?: GenericFileSystem,
  ): Promise<void> {
    // Persist the index store to disk.
  }
@@ -1,7 +1,7 @@
+import * as _ from "lodash";
 import * as path from "path";
 import { GenericFileSystem, exists } from "../FileSystem";
 import { DEFAULT_COLLECTION, DEFAULT_FS } from "../constants";
-import * as _ from "lodash";
 import { BaseKVStore } from "./types";

 export type DataType = Record<string, Record<string, any>>;
@@ -19,7 +19,7 @@ export class SimpleKVStore extends BaseKVStore {
  async put(
    key: string,
    val: any,
-    collection: string = DEFAULT_COLLECTION
+    collection: string = DEFAULT_COLLECTION,
  ): Promise<void> {
    if (!(collection in this.data)) {
      this.data[collection] = {};
@@ -33,7 +33,7 @@ export class SimpleKVStore extends BaseKVStore {

  async get(
    key: string,
-    collection: string = DEFAULT_COLLECTION
+    collection: string = DEFAULT_COLLECTION,
  ): Promise<any> {
    let collectionData = this.data[collection];
    if (_.isNil(collectionData)) {
@@ -51,7 +51,7 @@ export class SimpleKVStore extends BaseKVStore {

  async delete(
    key: string,
-    collection: string = DEFAULT_COLLECTION
+    collection: string = DEFAULT_COLLECTION,
  ): Promise<boolean> {
    if (key in this.data[collection]) {
      delete this.data[collection][key];
@@ -72,7 +72,7 @@ export class SimpleKVStore extends BaseKVStore {

  static async fromPersistPath(
    persistPath: string,
-    fs?: GenericFileSystem
+    fs?: GenericFileSystem,
  ): Promise<SimpleKVStore> {
    fs = fs || DEFAULT_FS;
    let dirPath = path.dirname(persistPath);
@@ -86,7 +86,7 @@ export class SimpleKVStore extends BaseKVStore {
      data = JSON.parse(fileData.toString());
    } catch (e) {
      console.error(
-        `No valid data found at path: ${persistPath} starting new store.`
+        `No valid data found at path: ${persistPath} starting new store.`,
      );
    }

@@ -7,7 +7,7 @@ export abstract class BaseKVStore {
  abstract put(
    key: string,
    val: Record<string, any>,
-    collection?: string
+    collection?: string,
  ): Promise<void>;
  abstract get(key: string, collection?: string): Promise<StoredValue>;
  abstract getAll(collection?: string): Promise<Record<string, StoredValue>>;
@@ -6,8 +6,8 @@ import {
  getTopKMMREmbeddings,
 } from "../../Embedding";
 import { BaseNode } from "../../Node";
+import { GenericFileSystem, exists } from "../FileSystem";
 import { DEFAULT_FS, DEFAULT_PERSIST_DIR } from "../constants";
-import { exists, GenericFileSystem } from "../FileSystem";
 import {
  VectorStore,
  VectorStoreQuery,
@@ -11,7 +11,7 @@ describe("similarity", () => {
    const embedding1 = [1, 2, 3];
    const embedding2 = [4, 5, 6];
    expect(() =>
-      similarity(embedding1, embedding2, "unknown" as SimilarityType)
+      similarity(embedding1, embedding2, "unknown" as SimilarityType),
    ).toThrow();
  });

@@ -19,7 +19,7 @@ describe("similarity", () => {
    const embedding1 = [1, 2, 3];
    const embedding2 = [4, 5, 6];
    expect(
-      similarity(embedding1, embedding2, SimilarityType.DOT_PRODUCT)
+      similarity(embedding1, embedding2, SimilarityType.DOT_PRODUCT),
    ).toEqual(32);
  });

@@ -27,7 +27,7 @@ describe("similarity", () => {
    const embedding1 = [1, 0];
    const embedding2 = [0, 1];
    expect(similarity(embedding1, embedding2, SimilarityType.DEFAULT)).toEqual(
-      0.0
+      0.0,
    );
  });

@@ -36,9 +36,9 @@ describe("similarity", () => {
    const docEmbedding1 = [0, 1]; // farther from query, distance 1.414
    const docEmbedding2 = [1, 1]; // closer to query distance 1
    expect(
-      similarity(queryEmbedding, docEmbedding1, SimilarityType.EUCLIDEAN)
+      similarity(queryEmbedding, docEmbedding1, SimilarityType.EUCLIDEAN),
    ).toBeLessThan(
-      similarity(queryEmbedding, docEmbedding2, SimilarityType.EUCLIDEAN)
+      similarity(queryEmbedding, docEmbedding2, SimilarityType.EUCLIDEAN),
    );
  });
 });
@@ -1,12 +1,12 @@
-import {
-  GenericFileSystem,
-  getNodeFS,
-  InMemoryFileSystem,
-  exists,
-  walk,
-} from "../storage/FileSystem";
 import os from "os";
 import path from "path";
+import {
+  GenericFileSystem,
+  InMemoryFileSystem,
+  exists,
+  getNodeFS,
+  walk,
+} from "../storage/FileSystem";

 type FileSystemUnderTest = {
  name: string;
@@ -61,7 +61,7 @@ describe.each<FileSystemUnderTest>([
    it("writes file to memory", async () => {
      await testFS.writeFile(`${tempDir}/test.txt`, "Hello, world!");
      expect(await testFS.readFile(`${tempDir}/test.txt`, "utf-8")).toBe(
-        "Hello, world!"
+        "Hello, world!",
      );
    });

@@ -69,7 +69,7 @@ describe.each<FileSystemUnderTest>([
      await testFS.writeFile(`${tempDir}/test.txt`, "Hello, world!");
      await testFS.writeFile(`${tempDir}/test.txt`, "Hello, again!");
      expect(await testFS.readFile(`${tempDir}/test.txt`, "utf-8")).toBe(
-        "Hello, again!"
+        "Hello, again!",
      );
    });
  });
@@ -77,7 +77,7 @@ describe.each<FileSystemUnderTest>([
  describe("readFile", () => {
    it("throws error for non-existing file", async () => {
      await expect(
-        testFS.readFile(`${tempDir}/not_exist.txt`, "utf-8")
+        testFS.readFile(`${tempDir}/not_exist.txt`, "utf-8"),
      ).rejects.toThrow();
    });
  });
@@ -0,0 +1,84 @@
+import {
+  rakeExtractKeywords,
+  simpleExtractKeywords,
+} from "../indices/keyword/utils";
+describe("SimpleExtractKeywords", () => {
+  test("should extract unique keywords", () => {
+    const text = "apple banana apple cherry";
+    const result = simpleExtractKeywords(text);
+    expect(result).toEqual(new Set(["apple", "banana", "cherry"]));
+  });
+
+  test("should handle empty string", () => {
+    const text = "";
+    const result = simpleExtractKeywords(text);
+    expect(result).toEqual(new Set());
+  });
+
+  test("should handle case sensitivity", () => {
+    const text = "Apple apple";
+    const result = simpleExtractKeywords(text);
+    expect(result).toEqual(new Set(["apple"]));
+  });
+
+  test("should order keywords by frequency", () => {
+    const text = "apple banana apple cherry banana apple";
+    const result = simpleExtractKeywords(text);
+    expect([...result]).toEqual(["apple", "banana", "cherry"]);
+  });
+
+  test("should respect the maxKeywords parameter", () => {
+    const text = "apple banana apple cherry banana apple orange";
+    const result = simpleExtractKeywords(text, 2);
+    expect(result).toEqual(new Set(["apple", "banana"]));
+  });
+
+  test("should handle non-alphabetic characters", () => {
+    const text = "apple! banana... apple? cherry, orange;";
+    const result = simpleExtractKeywords(text);
+    expect(result).toEqual(new Set(["apple", "banana", "cherry", "orange"]));
+  });
+});
+
+describe("RakeExtractKeywords", () => {
+  const sampleText = `Before college the two main things I worked on, outside of school, were writing and programming. I didn't write essays. I wrote what beginning writers were supposed to write then, and probably still are: short stories. My stories were awful. They had hardly any plot, just characters with strong feelings, which I imagined made them deep.`;
+  test("should return all keywords if maxKeywords is not provided", () => {
+    const result = rakeExtractKeywords(sampleText);
+    expect(result).toEqual(
+      new Set([
+        "strong feelings",
+        "beginning writers",
+        "short stories",
+        "write essays",
+        "stories",
+        "write",
+        "deep",
+        "imagined",
+        "characters",
+        "plot",
+        "hardly",
+        "awful",
+        "probably",
+        "supposed",
+        "wrote",
+        "didn",
+        "programming",
+        "writing",
+        "school",
+        "outside",
+        "main",
+        "college",
+      ]),
+    );
+  });
+
+  test("should respect the maxKeywords parameter", () => {
+    const result = rakeExtractKeywords(sampleText, 2);
+    expect(result).toEqual(new Set(["strong feelings", "beginning writers"]));
+  });
+
+  test("should handle empty return from rake", () => {
+    const result = rakeExtractKeywords("");
+    expect(result).toEqual(new Set());
+  });
+});
@@ -0,0 +1,90 @@
+import { SubQuestionOutputParser } from "../OutputParser";
+
+//This parser is really important, so make sure to add tests
+// as the parser sees through more iterations.
+describe("SubQuestionOutputParser", () => {
+  test("parses expected", () => {
+    const parser = new SubQuestionOutputParser();
+
+    const data = [
+      {
+        name: "uber_10k",
+        description: "Provides information about Uber financials for year 2021",
+      },
+      {
+        name: "lyft_10k",
+        description: "Provides information about Lyft financials for year 2021",
+      },
+    ];
+
+    const data_str: string = JSON.stringify(data);
+    const full_string = `\`\`\`json
+${data_str}
+\`\`\``;
+
+    const real_answer = { parsedOutput: data, rawOutput: full_string };
+
+    expect(parser.parse(full_string)).toEqual(real_answer);
+  });
+
+  //This is in case our LLM outputs a list response, but without ```json.
+  test("parses without ```json", () => {
+    const parser = new SubQuestionOutputParser();
+
+    const data = [
+      {
+        name: "uber_10k",
+        description: "Provides information about Uber financials for year 2021",
+      },
+      {
+        name: "lyft_10k",
+        description: "Provides information about Lyft financials for year 2021",
+      },
+    ];
+
+    const data_str: string = JSON.stringify(data);
+    const full_string = `${data_str}`;
+
+    const real_answer = { parsedOutput: data, rawOutput: full_string };
+
+    expect(parser.parse(JSON.stringify(data))).toEqual(real_answer);
+  });
+
+  test("parses null single response", () => {
+    const parser = new SubQuestionOutputParser();
+    const data_str =
+      "[\n" +
+      "    {\n" +
+      `        "subQuestion": "Sorry, I don't have any relevant information to answer your question",\n` +
+      '        "toolName": ""\n' +
+      "    }\n" +
+      "]";
+    const data = [
+      {
+        subQuestion:
+          "Sorry, I don't have any relevant information to answer your question",
+        toolName: "",
+      },
+    ];
+    const real_answer = { parsedOutput: data, rawOutput: data_str };
+    expect(parser.parse(data_str)).toEqual(real_answer);
+  });
+
+  test("Single JSON object case", () => {
+    const parser = new SubQuestionOutputParser();
+    const data_str =
+      "    {\n" +
+      `        "subQuestion": "Sorry, I don't have any relevant information to answer your question",\n` +
+      '        "toolName": ""\n' +
+      "    }\n";
+    const data = [
+      {
+        subQuestion:
+          "Sorry, I don't have any relevant information to answer your question",
+        toolName: "",
+      },
+    ];
+    const real_answer = { parsedOutput: data, rawOutput: data_str };
+    expect(parser.parse(data_str)).toEqual(real_answer);
+  });
+});
@@ -1,8 +1,5 @@
 import { existsSync, rmSync } from "fs";
-import {
-  storageContextFromDefaults,
-  StorageContext,
-} from "../storage/StorageContext";
+import { storageContextFromDefaults } from "../storage/StorageContext";

 jest.spyOn(console, "error");

@@ -1,7 +1,7 @@
 import { OpenAIEmbedding } from "../../Embedding";
 import { globalsHelper } from "../../GlobalsHelper";
-import { ChatMessage, OpenAI } from "../../llm/LLM";
 import { CallbackManager, Event } from "../../callbacks/CallbackManager";
+import { ChatMessage, OpenAI } from "../../llm/LLM";

 export function mockLlmGeneration({
  languageModel,
@@ -57,7 +57,7 @@ export function mockLlmGeneration({
            },
          });
        });
-      }
+      },
    );
 }

@@ -26,6 +26,9 @@ module.exports = {
          "DEBUG",
          "no_proxy",
          "NO_PROXY",
+
+          "NOTION_TOKEN",
+          "MONGODB_URI",
        ],
      },
    ],
@@ -2,11 +2,22 @@
 # For details on our config file, check out our docs at https://docs.sweep.dev

 # If you use this be sure to frequently sync your default branch(main, master) to dev.
-branch: 'main'
+branch: "main"
 # If you want to enable GitHub Actions for Sweep, set this to true.
 gha_enabled: False
 # This is the description of your project. It will be used by sweep when creating PRs. You can tell Sweep what's unique about your project, what frameworks you use, or anything else you want.
 # Here's an example: sweepai/sweep is a python project. The main api endpoints are in sweepai/api.py. Write code that adheres to PEP8.
-description: ''
+description: "LlamaIndexTS is a data framework in TypeScript for your LLM applications"
+
+sandbox:
+  install:
+    - npm install -g pnpm
+    - pnpm i
+    - pnpm add --save-dev prettier -w
+  check:
+    - pnpx prettier --write {file_path}
+    - pnpm eslint --fix {file_path}
+    - pnpx ts-node --type-check {file_path}
+    - pnpm test

 # Default Values: https://github.com/sweepai/sweep/blob/main/sweep.yaml
Author	SHA1	Message	Date
Yi Ding	837854de1e	rolled back notion package and changeset	2023-09-14 10:41:00 -07:00
yisding	8cc1f0726f	Merge pull request #112 from kkang2097/fix-output-parser Create OutputParser.test.ts [Needs Merge]	2023-09-14 09:32:35 -07:00
yisding	e1d617ef70	Merge pull request #122 from kevinlu1248/patch-1 Update sweep.yaml with newest sandbox format	2023-09-13 20:49:22 -07:00
Kevin Lu	5f199d68f9	Update sweep.yaml	2023-09-13 18:49:11 -07:00
Elliot Kang	b8cca2db97	make parseJsonMarkdown exportable	2023-09-13 15:30:55 -07:00
Elliot Kang	35e959219d	prettify OutputParser	2023-09-12 17:14:39 -07:00
Elliot Kang	08d466faee	Ported Python _marshall_llm_to_json From the Python side: - output_parsers/utils We'll still call this parseJsonMarkdown on the TS side	2023-09-12 17:13:02 -07:00
Yi Ding	0b5823f451	updated packages	2023-09-11 21:53:15 -07:00
Elliot Kang	c77b150c28	hardcoding single JSON object case	2023-09-11 15:24:07 -07:00
Elliot Kang	3cf27bb838	Okay, should be final version.	2023-09-11 14:55:50 -07:00
Elliot Kang	26a90435c7	Revert "Simplified OutputParser" This reverts commit `ff0e831da9`.	2023-09-11 14:55:07 -07:00
Elliot Kang	f6efaba906	Update OutputParser.test.ts - test cases are much simpler now.	2023-09-11 14:46:21 -07:00
Elliot Kang	ff0e831da9	Simplified OutputParser	2023-09-11 14:46:01 -07:00
Yi Ding	96bb65723a	changesets	2023-09-11 11:14:17 -07:00
Yi Ding	33ac4bc424	changelog typo	2023-09-11 10:45:27 -07:00
yisding	698503b467	Merge pull request #108 from run-llama/sweep/fix-broken-link-summary-index Fix broken link to Summary Index in end_to_end.md: Typo correction	2023-09-11 10:33:52 -07:00
yisding	0657525d40	Merge pull request #99 from kkang2097/main Add MongoReader [Needs merge]	2023-09-11 10:33:01 -07:00
Yi Ding	064d0de531	fix lint	2023-09-11 10:31:37 -07:00
Elliot Kang	471bf36a7b	Delete MongoReader.ts	2023-09-11 10:19:40 -07:00
Elliot Kang	cb7d2b4040	Revert "commit cleanup" This reverts commit 3dd334c6db5b211080e7a0b269e58e160914acc2.	2023-09-11 10:19:38 -07:00
Elliot Kang	6032cbcf45	Fixed typing to be more restrictive - Should have done this in the beginning, {Key:Value} objects should always be defined by Record<string, any>	2023-09-11 10:18:13 -07:00
Elliot Kang	73785d7552	Run Prettier, minor fixes -changed .limit(Infinity) to .limit(0) in readers/SimpleMongoReader.ts	2023-09-11 10:18:13 -07:00
Elliot Kang	431b5ffa59	rename to simpleMongoReader	2023-09-11 10:18:13 -07:00
Elliot Kang	c0500a0d4d	SimpleMongoReader demo	2023-09-11 10:18:13 -07:00
Yi Ding	5300534188	commit cleanup	2023-09-11 10:17:49 -07:00
Elliot Kang	02192a5f53	Create MongoReader.ts	2023-09-11 10:15:55 -07:00
Elliot Kang	2a98d5b8ee	Add MongoReader	2023-09-11 10:15:55 -07:00
yisding	4f495b5fc6	Merge pull request #110 from TomPenguin/extend-document-type Enhancing Type Safety for metadata	2023-09-11 10:12:39 -07:00
Elliot Kang	b75e2d23a2	re-ordering logic for parser - previous iteration ran the computation twice if we had an unexpected output format - added comment for future use	2023-09-11 01:16:31 -07:00
Elliot Kang	5261cdc794	Update OutputParser.test.ts removing comment	2023-09-10 21:52:57 -07:00
Elliot Kang	b179f61c6f	Update OutputParser.ts Essentially, we're giving OutputParser an option to parse List[JSON object] in case our LLM doesn't give us the exact output we want.	2023-09-10 21:39:41 -07:00
Elliot Kang	71b245ad6f	Update OutputParser.test.ts added new test cases, our LLM is not guaranteed to give us the exact formatted output.	2023-09-10 21:36:16 -07:00
Elliot Kang	5b070cf87a	Add new test, this one fails - fix this after adding all tests	2023-09-10 20:19:38 -07:00
Elliot Kang	b8afe0b364	Update OutputParser.test.ts - initial version of test script	2023-09-10 19:57:07 -07:00
Elliot Kang	92b4ec48f7	Create OutputParser.test.ts - SubQuestionOutputParser is not working as expected, writing some tests to check it out. QuestionGenerator outputs a list of objects in string format, which is unexpected. In particular: "[{prompt: "smth", response: "nothing"}]"	2023-09-10 05:00:09 -07:00
TomPenguin	6a69ac997d	Type-safe Metadata	2023-09-09 12:02:33 +09:00
Yi Ding	8c542c30a9	0.0.27	2023-09-08 08:49:27 -07:00
Yi Ding	5c59f93138	copy over example	2023-09-08 08:46:31 -07:00
Yi Ding	4a5591be75	changeset	2023-09-08 08:45:58 -07:00
yisding	e756764398	Merge pull request #100 from marcusschiesser/feat/summarize-chat-history feat: added draft for a summarizer of the chat history	2023-09-08 08:43:39 -07:00
yisding	568b9c3a4c	Merge pull request #98 from TomPenguin/add-support-notion-db Add Notion database support to NotionReader	2023-09-08 06:50:38 -07:00
yisding	13a4aa5212	Merge pull request #92 from swk777/larry/keyword support keyword index	2023-09-08 06:50:02 -07:00
sweep-ai[bot]	a8388c841f	Updated apps/docs/docs/end_to_end.md	2023-09-06 21:22:28 +00:00
Marcus Schiesser	87f1f59855	feat: added draft for a summarizer of the chat history	2023-09-05 14:00:47 +07:00
TomPenguin	c8c67d2a3d	add doc comments	2023-09-04 10:37:50 +09:00
TomPenguin	a042fa0b9a	add options	2023-09-04 10:14:50 +09:00
TomPenguin	b68d870599	support notion db by using notion-md-crawler	2023-09-04 09:03:42 +09:00
Yi Ding	2c63f10dca	0.0.26	2023-09-02 12:17:31 -07:00
yisding	7b8e2d0dc7	Merge pull request #94 from TomPenguin/add-notion-reader Add NotionReader	2023-09-02 12:16:16 -07:00
Yi Ding	5bb55bcc7d	changeset and bug fix	2023-09-02 12:14:30 -07:00
Yi Ding	08c49b0d5f	more demo work	2023-09-02 12:11:52 -07:00
Yi Ding	5b07c8adc6	beefed up notion example	2023-09-02 11:52:20 -07:00
TomPenguin	ca7e61c701	Mod import bug	2023-09-01 16:21:39 +09:00
TomPenguin	6795df10bd	Update md-utils-ts	2023-09-01 16:21:12 +09:00
swk777	1e98a35953	replace rake-pos	2023-09-01 15:08:19 +08:00
Yi Ding	2ab3fedf8f	adding test script	2023-08-31 22:50:27 -07:00
TomPenguin	915fc33dd7	Add NotionReader	2023-08-31 17:06:49 -07:00
Yi Ding	c79a5359b1	minor changes	2023-08-31 16:45:48 -07:00
swk777	632f176cdd	fix error	2023-08-31 16:24:13 -07:00
swk777	99e7857ac8	remove comment	2023-08-31 16:24:13 -07:00
swk777	9ff3837e49	add test case	2023-08-31 16:24:13 -07:00
swk777	6c91d0da5a	add example	2023-08-31 16:24:13 -07:00
swk777	12dd3c5eea	add keyword index and retriever	2023-08-31 16:24:11 -07:00
Yi Ding	04da822826	0.0.25	2023-08-30 13:09:54 -07:00
Yi Ding	40a8f0775e	updated exports for storage/index.ts	2023-08-30 12:06:58 -07:00
yisding	65aaebe2b5	Merge pull request #93 from andfk/patch-1 Update README.md for apps example	2023-08-30 06:20:40 -07:00
Andrés F	769559279f	Update README.md The commands should be inverted	2023-08-30 18:13:00 +08:00
Yi Ding	bb7fd38c46	removing the newline to space embedding conversion https://github.com/openai/openai-python/issues/418	2023-08-29 22:35:41 -07:00
Yi Ding	a734927a42	upgrade prettier and prettiered	2023-08-29 20:30:02 -07:00
Yi Ding	e21eca2a16	updated packages and README	2023-08-29 20:24:29 -07:00
Yi Ding	33c8c2fe47	documentation update for SummaryIndex	2023-08-29 14:06:12 -07:00
Yi Ding	c3048858e9	0.0.24	2023-08-29 12:34:21 -07:00