Release 0.5.16 (#1115)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

.eslintrc.cjs

@@ -31,7 +31,12 @@ module.exports = {
     "@typescript-eslint/ban-types": "off",
     "no-array-constructor": "off",
     "@typescript-eslint/no-array-constructor": "off",
-    "@typescript-eslint/no-base-to-string": "off",
+    "@typescript-eslint/no-base-to-string": [
+      "error",
+      {
+        ignoredTypeNames: ["Error", "RegExp", "URL", "URLSearchParams"],
+      },
+    ],
     "@typescript-eslint/no-duplicate-enum-values": "off",
     "@typescript-eslint/no-duplicate-type-constituents": "off",
     "@typescript-eslint/no-explicit-any": "off",

CONTRIBUTING.md

@@ -4,11 +4,11 @@
 
 This is a monorepo built with Turborepo
 
-Right now there are two packages of importance:
+Right now, for first-time contributors, these three packages are of the highest importance:
 
-packages/llamaindex which is the main NPM library llamaindex
-
-examples is where the demo code lives
+- `packages/llamaindex` which is the main NPM library `llamaindex`
+- `examples` is where the demo code lives
+- `apps/docs` is where the code for the documentation of https://ts.llamaindex.ai/ is located
 
 ### Turborepo docs
 
@@ -43,11 +43,11 @@ pnpm run test
 
 To write new test cases write them in [packages/llamaindex/tests](/packages/llamaindex/tests)
 
-We use Jest https://jestjs.io/ to write our test cases. Jest comes with a bunch of built in assertions using the expect function: https://jestjs.io/docs/expect
+We use Vitest https://vitest.dev to write our test cases. Vitest comes with a bunch of built-in assertions using the expect function: https://vitest.dev/api/expect.html#expect
 
 ### Demo applications
 
-There is an existing ["example"](/examples/README.md) demos folder with mainly NodeJS scripts. Feel free to add additional demos to that folder. If you would like to try out your changes in the core package with a new demo, you need to run the build command in the README.
+There is an existing ["example"](/examples/README.md) demos folder with mainly NodeJS scripts. Feel free to add additional demos to that folder. If you would like to try out your changes in the `llamaindex` package with a new demo, you need to run the build command in the README.
 
 You can create new demo applications in the apps folder. Just run pnpm init in the folder after you create it to create its own package.json
 
@@ -81,7 +81,7 @@ Any changes you make should be reflected in the browser. If you need to regenera
 
 ## Changeset
 
-We use [changesets](https://github.com/changesets/changesets) for managing versions and changelogs. To create a new changeset, run:
+We use [changesets](https://github.com/changesets/changesets) for managing versions and changelogs. To create a new changeset, run in the root folder:
 
 ```
 pnpm changeset

README.md

@@ -164,7 +164,7 @@ Check out our NextJS playground at https://llama-playground.vercel.app/. The sou
 
 - [Node](/packages/llamaindex/src/Node.ts): The basic data building block. Most commonly, these are parts of the document split into manageable pieces that are small enough to be fed into an embedding model and LLM.
 
-- [Embedding](/packages/llamaindex/src/embeddings/OpenAIEmbedding.ts): Embeddings are sets of floating point numbers which represent the data in a Node. By comparing the similarity of embeddings, we can derive an understanding of the similarity of two pieces of data. One use case is to compare the embedding of a question with the embeddings of our Nodes to see which Nodes may contain the data needed to answer that quesiton. Because the default service context is OpenAI, the default embedding is `OpenAIEmbedding`. If using different models, say through Ollama, use this [Embedding](/packages/llamaindex/src/embeddings/OllamaEmbedding.ts) (see all [here](/packages/llamaindex/src/embeddings)).
+- [Embedding](/packages/llamaindex/src/embeddings/OpenAIEmbedding.ts): Embeddings are sets of floating point numbers which represent the data in a Node. By comparing the similarity of embeddings, we can derive an understanding of the similarity of two pieces of data. One use case is to compare the embedding of a question with the embeddings of our Nodes to see which Nodes may contain the data needed to answer that question. Because the default service context is OpenAI, the default embedding is `OpenAIEmbedding`. If using different models, say through Ollama, use this [Embedding](/packages/llamaindex/src/embeddings/OllamaEmbedding.ts) (see all [here](/packages/llamaindex/src/embeddings)).
 
 - [Indices](/packages/llamaindex/src/indices/): Indices store the Nodes and the embeddings of those nodes. QueryEngines retrieve Nodes from these Indices using embedding similarity.
 

apps/docs/.changeset/tall-parents-add.md

@@ -0,0 +1,6 @@
+---
+"llamaindex": minor
+"docs": minor
+---
+
+Add deepseek llm class

apps/docs/CHANGELOG.md

@@ -1,5 +1,117 @@
 # docs
 
+## 0.0.57
+
+### Patch Changes
+
+- Updated dependencies [58abc57]
+  - llamaindex@0.5.16
+
+## 0.0.56
+
+### Patch Changes
+
+- Updated dependencies [01c184c]
+- Updated dependencies [07a275f]
+  - llamaindex@0.5.15
+
+## 0.0.55
+
+### Patch Changes
+
+- Updated dependencies [c825a2f]
+  - llamaindex@0.5.14
+
+## 0.0.54
+
+### Patch Changes
+
+- llamaindex@0.5.13
+
+## 0.0.53
+
+### Patch Changes
+
+- Updated dependencies [345300f]
+- Updated dependencies [da5cfc4]
+- Updated dependencies [da5cfc4]
+  - llamaindex@0.5.12
+
+## 0.0.52
+
+### Patch Changes
+
+- 376d29a: feat: added tool calling and agent support for llama3.1 504B
+  - llamaindex@0.5.11
+
+## 0.0.51
+
+### Patch Changes
+
+- 086b940: feat: add DeepSeek LLM
+- 5d5716b: feat: add a reader for JSON data
+- Updated dependencies [086b940]
+- Updated dependencies [5d5716b]
+- Updated dependencies [91d02a4]
+- Updated dependencies [fb6db45]
+  - llamaindex@0.5.10
+
+## 0.0.50
+
+### Patch Changes
+
+- Updated dependencies [15962b3]
+  - llamaindex@0.5.9
+
+## 0.0.49
+
+### Patch Changes
+
+- Updated dependencies [3d5ba08]
+- Updated dependencies [d917cdc]
+  - llamaindex@0.5.8
+
+## 0.0.48
+
+### Patch Changes
+
+- Updated dependencies [ec59acd]
+  - llamaindex@0.5.7
+
+## 0.0.47
+
+### Patch Changes
+
+- Updated dependencies [2562244]
+- Updated dependencies [325aa51]
+- Updated dependencies [ab700ea]
+- Updated dependencies [92f0782]
+- Updated dependencies [6cf6ae6]
+- Updated dependencies [b7cfe5b]
+  - llamaindex@0.5.6
+
+## 0.0.46
+
+### Patch Changes
+
+- Updated dependencies [b974eea]
+  - llamaindex@0.5.5
+
+## 0.0.45
+
+### Patch Changes
+
+- Updated dependencies [1a65ead]
+  - llamaindex@0.5.4
+
+## 0.0.44
+
+### Patch Changes
+
+- Updated dependencies [9bbbc67]
+- Updated dependencies [b3681bf]
+  - llamaindex@0.5.3
+
 ## 0.0.43
 
 ### Patch Changes

apps/docs/docs/modules/agent/index.md

@@ -15,6 +15,7 @@ LlamaIndex.TS comes with a few built-in agents, but you can also create your own
 - Anthropic Agent both via Anthropic and Bedrock (in `@llamaIndex/community`)
 - Gemini Agent
 - ReACT Agent
+- Meta3.1 504B via Bedrock (in `@llamaIndex/community`)
 
 ## Examples
 

apps/docs/docs/modules/data_loaders/discord.mdx

@@ -0,0 +1,34 @@
+import CodeBlock from "@theme/CodeBlock";
+import CodeSource from "!raw-loader!../../../../../examples/readers/src/discord";
+
+# DiscordReader
+
+DiscordReader is a simple data loader that reads all messages in a given Discord channel and returns them as Document objects.
+It uses the [@discordjs/rest](https://github.com/discordjs/discord.js/tree/main/packages/rest) library to fetch the messages.
+
+## Usage
+
+First step is to create a Discord Application and generating a bot token [here](https://discord.com/developers/applications).
+In your Discord Application, go to the `OAuth2` tab and generate an invite URL by selecting `bot` and click `Read Messages/View Channels` as wells as `Read Message History`.
+This will invite the bot with the necessary permissions to read messages.
+Copy the URL in your browser and select the server you want your bot to join.
+
+<CodeBlock language="ts">{CodeSource}</CodeBlock>
+
+### Params
+
+#### DiscordReader()
+
+- `discordToken?`: The Discord bot token.
+- `requestHandler?`: Optionally provide a custom request function for edge environments, e.g. `fetch`. See discord.js for more info.
+
+#### DiscordReader.loadData
+
+- `channelIDs`: The ID(s) of discord channels as an array of strings.
+- `limit?`: Optionally limit the number of messages to read
+- `additionalInfo?`: An optional flag to include embedded messages and attachment urls in the document.
+- `oldestFirst?`: An optional flag to return the oldest messages first.
+
+## API Reference
+
+- [DiscordReader](../../api/classes/DiscordReader.md)

apps/docs/docs/modules/data_loaders/index.mdx

@@ -16,7 +16,15 @@ It is a simple reader that reads all files from a directory and its subdirectori
 
 <CodeBlock language="ts">{CodeSource}</CodeBlock>
 
-Currently, it supports reading `.txt`, `.pdf`, `.csv`, `.md`, `.docx`, `.htm`, `.html`, `.jpg`, `.jpeg`, `.png` and `.gif` files, but support for other file types is planned.
+Currently, the following readers are mapped to specific file types:
+
+- [TextFileReader](../../api/classes/TextFileReader.md): `.txt`
+- [PDFReader](../../api/classes/PDFReader.md): `.pdf`
+- [PapaCSVReader](../../api/classes/PapaCSVReader.md): `.csv`
+- [MarkdownReader](../../api/classes/MarkdownReader.md): `.md`
+- [DocxReader](../../api/classes/DocxReader.md): `.docx`
+- [HTMLReader](../../api/classes/HTMLReader.md): `.htm`, `.html`
+- [ImageReader](../../api/classes/ImageReader.md): `.jpg`, `.jpeg`, `.png`, `.gif`
 
 You can modify the reader three different ways:
 

apps/docs/docs/modules/data_loaders/json.md

@@ -0,0 +1,144 @@
+# JSONReader
+
+A simple JSON data loader with various options.
+Either parses the entire string, cleaning it and treat each line as an embedding or performs a recursive depth-first traversal yielding JSON paths.
+
+## Usage
+
+```ts
+import { JSONReader } from "llamaindex";
+
+const file = "../../PATH/TO/FILE";
+const content = new TextEncoder().encode("JSON_CONTENT");
+
+const reader = new JSONReader({ levelsBack: 0, collapseLength: 100 });
+const docsFromFile = reader.loadData(file);
+const docsFromContent = reader.loadDataAsContent(content);
+```
+
+### Options
+
+Basic:
+
+- `ensureAscii?`: Wether to ensure only ASCII characters be present in the output by converting non-ASCII characters to their unicode escape sequence. Default is `false`.
+
+- `isJsonLines?`: Wether the JSON is in JSON Lines format. If true, will split into lines, remove empty one and parse each line as JSON. Default is `false`
+
+- `cleanJson?`: Whether to clean the JSON by filtering out structural characters (`{}, [], and ,`). If set to false, it will just parse the JSON, not removing structural characters. Default is `true`.
+
+Depth-First-Traversal:
+
+- `levelsBack?`: Specifies how many levels up the JSON structure to include in the output. `cleanJson` will be ignored. If set to 0, all levels are included. If undefined, parses the entire JSON, treat each line as an embedding and create a document per top-level array. Default is `undefined`
+
+- `collapseLength?`: The maximum length of JSON string representation to be collapsed into a single line. Only applicable when `levelsBack` is set. Default is `undefined`
+
+#### Examples
+
+<!-- prettier-ignore-start -->
+Input:
+
+```json
+{"a": {"1": {"key1": "value1"}, "2": {"key2": "value2"}}, "b": {"3": {"k3": "v3"}, "4": {"k4": "v4"}}}
+```
+
+Default options:
+
+`LevelsBack` = `undefined` & `cleanJson` = `true`
+
+Output:
+
+```json
+"a": {
+"1": {
+"key1": "value1"
+"2": {
+"key2": "value2"
+"b": {
+"3": {
+"k3": "v3"
+"4": {
+"k4": "v4"
+```
+
+Depth-First Traversal all levels:
+
+`levelsBack` = `0`
+
+Output:
+
+```json
+a 1 key1 value1
+a 2 key2 value2
+b 3 k3 v3
+b 4 k4 v4
+```
+
+Depth-First Traversal and Collapse:
+
+`levelsBack` = `0` & `collapseLength` = `35`
+
+Output:
+
+```json
+a 1 {"key1":"value1"}
+a 2 {"key2":"value2"}
+b {"3":{"k3":"v3"},"4":{"k4":"v4"}}
+```
+
+Depth-First Traversal limited levels:
+
+`levelsBack` = `2`
+
+Output:
+
+```json
+1 key1 value1
+2 key2 value2
+3 k3 v3
+4 k4 v4
+```
+
+Uncleaned JSON:
+
+`levelsBack` = `undefined` & `cleanJson` = `false`
+
+Output:
+
+```json
+{"a":{"1":{"key1":"value1"},"2":{"key2":"value2"}},"b":{"3":{"k3":"v3"},"4":{"k4":"v4"}}}
+```
+
+ASCII-Conversion:
+
+Input:
+
+```json
+{ "message": "こんにちは世界" }
+```
+
+Output:
+
+```json
+"message": "\u3053\u3093\u306b\u3061\u306f\u4e16\u754c"
+```
+
+JSON Lines Format:
+
+Input:
+
+```json
+{"tweet": "Hello world"}\n{"tweet": "こんにちは世界"}
+```
+
+Output:
+
+```json
+"tweet": "Hello world"
+
+"tweet": "こんにちは世界"
+```
+<!-- prettier-ignore-end -->
+
+## API Reference
+
+- [JSONReader](../../api/classes/JSONReader.md)

apps/docs/docs/modules/data_loaders/llama_parse/index.mdx

@@ -41,11 +41,17 @@ They can be divided into two groups.
 - `doNotCache?` Optional. Set to true to not cache the document.
 - `fastMode?` Optional. Set to true to use the fast mode. This mode will skip OCR of images, and table/heading reconstruction. Note: Non-compatible with `gpt4oMode`.
 - `doNotUnrollColumns?` Optional. Set to true to keep the text according to document layout. Reduce reconstruction accuracy, and LLMs/embeddings performances in most cases.
-- `pageSeparator?` Optional. The page separator to use. Defaults is `\\n---\\n`.
-- `gpt4oMode` set to true to use GPT-4o to extract content. Default is `false`.
-- `gpt4oApiKey?` Optional. Set the GPT-4o API key. Lowers the cost of parsing by using your own API key. Your OpenAI account will be charged. Can also be set in the environment variable `LLAMA_CLOUD_GPT4O_API_KEY`.
+- `pageSeparator?` Optional. A templated page separator to use to split the text. If the results contain `{page_number}` (e.g. JSON mode), it will be replaced by the next page number. If not set the default separator `\\n---\\n` will be used.
+- `pagePrefix?` Optional. A templated prefix to add to the beginning of each page. If the results contain `{page_number}`, it will be replaced by the page number.
+- `pageSuffix?` Optional. A templated suffix to add to the end of each page. If the results contain `{page_number}`, it will be replaced by the page number.
+- `gpt4oMode` Deprecated. Use vendorMultimodal params. Set to true to use GPT-4o to extract content. Default is `false`.
+- `gpt4oApiKey?` Deprecated. Use vendorMultimodal params. Optional. Set the GPT-4o API key. Lowers the cost of parsing by using your own API key. Your OpenAI account will be charged. Can also be set in the environment variable `LLAMA_CLOUD_GPT4O_API_KEY`.
 - `boundingBox?` Optional. Specify an area of the document to parse. Expects the bounding box margins as a string in clockwise order, e.g. `boundingBox = "0.1,0,0,0"` to not parse the top 10% of the document.
 - `targetPages?` Optional. Specify which pages to parse by specifying them as a comma-separated list. First page is `0`.
+- `splitByPage` Wether to split the results, creating one document per page. Uses the set `pageSeparator` or `\n---\n` as fallback. Default is true.
+- `useVendorMultimodalModel` set to true to use a multimodal model. Default is `false`.
+- `vendorMultimodalModel?` Optional. Specify which multimodal model to use. Default is GPT4o. See [here](https://docs.cloud.llamaindex.ai/llamaparse/features/multimodal) for a list of available models and cost.
+- `vendorMultimodalApiKey?` Optional. Set the multimodal model API key. Can also be set in the environment variable `LLAMA_CLOUD_VENDOR_MULTIMODAL_API_KEY`.
 - `numWorkers` as in the python version, is set in `SimpleDirectoryReader`. Default is 1.
 
 ### LlamaParse with SimpleDirectoryReader

apps/docs/docs/modules/llms/available_llms/bedrock.md

@@ -29,6 +29,9 @@ META_LLAMA2_13B_CHAT = "meta.llama2-13b-chat-v1";
 META_LLAMA2_70B_CHAT = "meta.llama2-70b-chat-v1";
 META_LLAMA3_8B_INSTRUCT = "meta.llama3-8b-instruct-v1:0";
 META_LLAMA3_70B_INSTRUCT = "meta.llama3-70b-instruct-v1:0";
+META_LLAMA3_1_8B_INSTRUCT = "meta.llama3-1-8b-instruct-v1:0"; // available on us-west-2
+META_LLAMA3_1_70B_INSTRUCT = "meta.llama3-1-70b-instruct-v1:0"; // available on us-west-2
+META_LLAMA3_1_405B_INSTRUCT = "meta.llama3-1-405b-instruct-v1:0"; // preview only, available on us-west-2, tool calling supported
 ```
 
 Sonnet, Haiku and Opus are multimodal, image_url only supports base64 data url format, e.g. `data:image/jpeg;base64,SGVsbG8sIFdvcmxkIQ==`
@@ -64,3 +67,72 @@ async function main() {
   console.log(response.response);
 }
 ```
+
+## Agent Example
+
+```ts
+import { BEDROCK_MODELS, Bedrock } from "@llamaindex/community";
+import { FunctionTool, LLMAgent } from "llamaindex";
+
+const sumNumbers = FunctionTool.from(
+  ({ a, b }: { a: number; b: number }) => `${a + b}`,
+  {
+    name: "sumNumbers",
+    description: "Use this function to sum two numbers",
+    parameters: {
+      type: "object",
+      properties: {
+        a: {
+          type: "number",
+          description: "The first number",
+        },
+        b: {
+          type: "number",
+          description: "The second number",
+        },
+      },
+      required: ["a", "b"],
+    },
+  },
+);
+
+const divideNumbers = FunctionTool.from(
+  ({ a, b }: { a: number; b: number }) => `${a / b}`,
+  {
+    name: "divideNumbers",
+    description: "Use this function to divide two numbers",
+    parameters: {
+      type: "object",
+      properties: {
+        a: {
+          type: "number",
+          description: "The dividend a to divide",
+        },
+        b: {
+          type: "number",
+          description: "The divisor b to divide by",
+        },
+      },
+      required: ["a", "b"],
+    },
+  },
+);
+
+const bedrock = new Bedrock({
+  model: BEDROCK_MODELS.META_LLAMA3_1_405B_INSTRUCT,
+  ...
+});
+
+async function main() {
+  const agent = new LLMAgent({
+    llm: bedrock,
+    tools: [sumNumbers, divideNumbers],
+  });
+
+  const response = await agent.chat({
+    message: "How much is 5 + 5? then divide by 2",
+  });
+
+  console.log(response.message);
+}
+```

apps/docs/docs/modules/llms/available_llms/deepseek.md

@@ -0,0 +1,50 @@
+# DeepSeek LLM
+
+## Usage
+
+```ts
+import { DeepSeekLLM, Settings } from "llamaindex";
+
+Settings.llm = new DeepSeekLLM({
+  apiKey: "<YOUR_API_KEY>",
+  model: "deepseek-coder", // or "deepseek-chat"
+});
+```
+
+## Example
+
+```ts
+import { DeepSeekLLM, Document, VectorStoreIndex, Settings } from "llamaindex";
+
+const deepseekLlm = new DeepSeekLLM({
+  apiKey: "<YOUR_API_KEY>",
+  model: "deepseek-coder", // or "deepseek-chat"
+});
+
+async function main() {
+  const response = await llm.deepseekLlm.chat({
+    messages: [
+      {
+        role: "system",
+        content: "You are an AI assistant",
+      },
+      {
+        role: "user",
+        content: "Tell me about San Francisco",
+      },
+    ],
+    stream: false,
+  });
+  console.log(response);
+}
+```
+
+# Limitations
+
+Currently does not support function calling.
+
+[Currently does not support json-output param while still is very good at json generating.](https://platform.deepseek.com/api-docs/faq#does-your-api-support-json-output)
+
+## API platform
+
+- [DeepSeek platform](https://platform.deepseek.com/)

apps/docs/docs/modules/node_postprocessors/cohere_reranker.md

@@ -39,8 +39,9 @@ const index = await VectorStoreIndex.fromDocuments([document]);
 The default value for `similarityTopK` is 2. This means that only the most similar document will be returned. To retrieve more results, you can increase the value of `similarityTopK`.
 
 ```ts
-const retriever = index.asRetriever();
-retriever.similarityTopK = 5;
+const retriever = index.asRetriever({
+  similarityTopK: 5,
+});
 ```
 
 ## Create a new instance of the CohereRerank class

apps/docs/docs/modules/node_postprocessors/jinaai_reranker.md

@@ -39,8 +39,9 @@ const index = await VectorStoreIndex.fromDocuments([document]);
 The default value for `similarityTopK` is 2. This means that only the most similar document will be returned. To retrieve more results, you can increase the value of `similarityTopK`.
 
 ```ts
-const retriever = index.asRetriever();
-retriever.similarityTopK = 5;
+const retriever = index.asRetriever({
+  similarityTopK: 5,
+});
 ```
 
 ## Create a new instance of the JinaAIReranker class

apps/docs/docs/modules/node_postprocessors/mixedbreadiai_reranker.md

@@ -55,8 +55,9 @@ const index = await VectorStoreIndex.fromDocuments([document]);
 The default value for `similarityTopK` is 2, which means only the most similar document will be returned. To get more results, like picking a variety of fresh breads, you can increase the value of `similarityTopK`.
 
 ```ts
-const retriever = index.asRetriever();
-retriever.similarityTopK = 5;
+const retriever = index.asRetriever({
+  similarityTopK: 5,
+});
 ```
 
 ### Step 3: Create a MixedbreadAIReranker Instance

apps/docs/docs/modules/query_engines/metadata_filtering.md

@@ -75,7 +75,7 @@ const queryEngine = index.asQueryEngine({
       {
         key: "dogId",
         value: "2",
-        filterType: "ExactMatch",
+        operator: "==",
       },
     ],
   },
@@ -88,6 +88,8 @@ const response = await queryEngine.query({
 console.log(response.toString());
 ```
 
+Besides using the equal operator (`==`), you can also use a whole set of different [operators](../../api/interfaces/MetadataFilter.md#operator) to filter your documents.
+
 ## Full Code
 
 ```ts
@@ -135,7 +137,7 @@ async function main() {
           {
             key: "dogId",
             value: "2",
-            filterType: "ExactMatch",
+            operator: "==",
           },
         ],
       },
@@ -156,3 +158,4 @@ main();
 
 - [VectorStoreIndex](../../api/classes/VectorStoreIndex.md)
 - [ChromaVectorStore](../../api/classes/ChromaVectorStore.md)
+- [MetadataFilter](../../api/interfaces/MetadataFilter.md)

apps/docs/docs/modules/retriever.md

@@ -7,8 +7,9 @@ sidebar_position: 5
 A retriever in LlamaIndex is what is used to fetch `Node`s from an index using a query string. Aa `VectorIndexRetriever` will fetch the top-k most similar nodes. Meanwhile, a `SummaryIndexRetriever` will fetch all nodes no matter the query.
 
 ```typescript
-const retriever = vector_index.asRetriever();
-retriever.similarityTopK = 3;
+const retriever = vectorIndex.asRetriever({
+  similarityTopK: 3,
+});
 
 // Fetch nodes!
 const nodesWithScore = await retriever.retrieve({ query: "query string" });

apps/docs/package.json

@@ -1,6 +1,6 @@
 {
   "name": "docs",
-  "version": "0.0.43",
+  "version": "0.0.57",
   "private": true,
   "scripts": {
     "docusaurus": "docusaurus",
@@ -34,10 +34,10 @@
     "@docusaurus/types": "3.4.0",
     "@tsconfig/docusaurus": "2.0.3",
     "@types/node": "^20.12.11",
-    "docusaurus-plugin-typedoc": "1.0.2",
-    "typedoc": "0.26.3",
+    "docusaurus-plugin-typedoc": "1.0.3",
+    "typedoc": "0.26.4",
     "typedoc-plugin-markdown": "4.1.2",
-    "typescript": "^5.5.2"
+    "typescript": "^5.5.3"
   },
   "browserslist": {
     "production": [

examples/chatEngine.ts

@@ -16,8 +16,9 @@ Settings.chunkSize = 512;
 async function main() {
   const document = new Document({ text: essay });
   const index = await VectorStoreIndex.fromDocuments([document]);
-  const retriever = index.asRetriever();
-  retriever.similarityTopK = 5;
+  const retriever = index.asRetriever({
+    similarityTopK: 5,
+  });
   const chatEngine = new ContextChatEngine({ retriever });
   const rl = readline.createInterface({ input, output });
 

examples/chromadb/preFilters.ts

@@ -40,7 +40,7 @@ async function main() {
           {
             key: "dogId",
             value: "2",
-            filterType: "ExactMatch",
+            operator: "==",
           },
         ],
       },

examples/longText.ts

@@ -1,17 +1,16 @@
 import {
   Document,
+  SentenceSplitter,
   Settings,
-  SimpleNodeParser,
   VectorStoreIndex,
 } from "llamaindex";
 
 export const STORAGE_DIR = "./data";
 
 // Update node parser
-Settings.nodeParser = new SimpleNodeParser({
+Settings.nodeParser = new SentenceSplitter({
   chunkSize: 512,
   chunkOverlap: 20,
-  splitLongSentences: true,
 });
 (async () => {
   // generate a document with a very long sentence (9000 words long)

examples/lowlevel.ts

@@ -27,11 +27,13 @@ import {
     },
   ];
 
-  const stream = await responseSynthesizer.synthesize({
-    query: "What age am I?",
-    nodesWithScore,
-    stream: true,
-  });
+  const stream = await responseSynthesizer.synthesize(
+    {
+      query: "What age am I?",
+      nodesWithScore,
+    },
+    true,
+  );
   for await (const chunk of stream) {
     process.stdout.write(chunk.response);
   }

examples/metadata-filter/milvus.ts

@@ -0,0 +1,40 @@
+import { MilvusVectorStore, VectorStoreIndex } from "llamaindex";
+
+const collectionName = "movie_reviews";
+
+async function main() {
+  try {
+    const milvus = new MilvusVectorStore({ collection: collectionName });
+    const index = await VectorStoreIndex.fromVectorStore(milvus);
+    const retriever = index.asRetriever({ similarityTopK: 20 });
+
+    console.log("\n=====\nQuerying the index with filters");
+    const queryEngineWithFilters = index.asQueryEngine({
+      retriever,
+      preFilters: {
+        filters: [
+          {
+            key: "document_id",
+            value: "./data/movie_reviews.csv_37",
+            operator: "==",
+          },
+          {
+            key: "document_id",
+            value: "./data/movie_reviews.csv_37",
+            operator: "!=",
+          },
+        ],
+        condition: "or",
+      },
+    });
+    const resultAfterFilter = await queryEngineWithFilters.query({
+      query: "Get all movie titles.",
+    });
+    console.log(`Query from ${resultAfterFilter.sourceNodes?.length} nodes`);
+    console.log(resultAfterFilter.response);
+  } catch (e) {
+    console.error(e);
+  }
+}
+
+void main();

examples/metadata-filter/simple.ts

@@ -0,0 +1,143 @@
+import {
+  Document,
+  Settings,
+  SimpleDocumentStore,
+  VectorStoreIndex,
+  storageContextFromDefaults,
+} from "llamaindex";
+
+Settings.callbackManager.on("retrieve-end", (event) => {
+  const { nodes } = event.detail;
+  console.log("Number of retrieved nodes:", nodes.length);
+});
+
+async function getDataSource() {
+  const docs = [
+    new Document({
+      text: "The dog is brown",
+      metadata: {
+        dogId: "1",
+        private: true,
+      },
+    }),
+    new Document({
+      text: "The dog is yellow",
+      metadata: {
+        dogId: "2",
+        private: false,
+      },
+    }),
+    new Document({
+      text: "The dog is red",
+      metadata: {
+        dogId: "3",
+        private: false,
+      },
+    }),
+  ];
+  const storageContext = await storageContextFromDefaults({
+    persistDir: "./cache",
+  });
+  const numberOfDocs = Object.keys(
+    (storageContext.docStore as SimpleDocumentStore).toDict(),
+  ).length;
+  if (numberOfDocs === 0) {
+    // Generate the data source if it's empty
+    return await VectorStoreIndex.fromDocuments(docs, {
+      storageContext,
+    });
+  }
+  return await VectorStoreIndex.init({
+    storageContext,
+  });
+}
+
+async function main() {
+  const index = await getDataSource();
+  console.log(
+    "=============\nQuerying index with no filters. The output should be any color.",
+  );
+  const queryEngineNoFilters = index.asQueryEngine({
+    similarityTopK: 3,
+  });
+  const noFilterResponse = await queryEngineNoFilters.query({
+    query: "What is the color of the dog?",
+  });
+  console.log("No filter response:", noFilterResponse.toString());
+
+  console.log(
+    "\n=============\nQuerying index with dogId 2 and private false. The output always should be red.",
+  );
+  const queryEngineEQ = index.asQueryEngine({
+    preFilters: {
+      filters: [
+        {
+          key: "private",
+          value: "false",
+          operator: "==",
+        },
+        {
+          key: "dogId",
+          value: "3",
+          operator: "==",
+        },
+      ],
+    },
+    similarityTopK: 3,
+  });
+  const responseEQ = await queryEngineEQ.query({
+    query: "What is the color of the dog?",
+  });
+  console.log("Filter with dogId 2 response:", responseEQ.toString());
+
+  console.log(
+    "\n=============\nQuerying index with dogId IN (1, 3). The output should be brown and red.",
+  );
+  const queryEngineIN = index.asQueryEngine({
+    preFilters: {
+      filters: [
+        {
+          key: "dogId",
+          value: ["1", "3"],
+          operator: "in",
+        },
+      ],
+    },
+    similarityTopK: 3,
+  });
+  const responseIN = await queryEngineIN.query({
+    query: "What is the color of the dog?",
+  });
+  console.log("Filter with dogId IN (1, 3) response:", responseIN.toString());
+
+  console.log(
+    "\n=============\nQuerying index with dogId IN (1, 3). The output should be any.",
+  );
+  const queryEngineOR = index.asQueryEngine({
+    preFilters: {
+      filters: [
+        {
+          key: "private",
+          value: "false",
+          operator: "==",
+        },
+        {
+          key: "dogId",
+          value: ["1", "3"],
+          operator: "in",
+        },
+      ],
+      condition: "or",
+    },
+    similarityTopK: 3,
+  });
+  const responseOR = await queryEngineOR.query({
+    query: "What is the color of the dog?",
+  });
+  console.log(
+    "Filter with dogId with OR operator response:",
+    responseOR.toString(),
+  );
+}
+
+void main();

examples/multimodal/jina.ts

@@ -0,0 +1,83 @@
+import {
+  ImageDocument,
+  JinaAIEmbedding,
+  similarity,
+  SimilarityType,
+  SimpleDirectoryReader,
+} from "llamaindex";
+import path from "path";
+
+async function main() {
+  const jina = new JinaAIEmbedding({
+    model: "jina-clip-v1",
+  });
+
+  // Get text embeddings
+  const text1 = "a car";
+  const textEmbedding1 = await jina.getTextEmbedding(text1);
+  const text2 = "a football match";
+  const textEmbedding2 = await jina.getTextEmbedding(text2);
+
+  // Get image embedding
+  const image =
+    "https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/football-match.jpg";
+  const imageEmbedding = await jina.getImageEmbedding(image);
+
+  // Calc similarity between text and image
+  const sim1 = similarity(
+    textEmbedding1,
+    imageEmbedding,
+    SimilarityType.DEFAULT,
+  );
+  const sim2 = similarity(
+    textEmbedding2,
+    imageEmbedding,
+    SimilarityType.DEFAULT,
+  );
+
+  console.log(`Similarity between "${text1}" and the image is ${sim1}`);
+  console.log(`Similarity between "${text2}" and the image is ${sim2}`);
+
+  // Get multiple text embeddings
+  const textEmbeddings = await jina.getTextEmbeddings([text1, text2]);
+  const sim3 = similarity(
+    textEmbeddings[0],
+    textEmbeddings[1],
+    SimilarityType.DEFAULT,
+  );
+  console.log(
+    `Similarity between the two texts "${text1}" and "${text2}" is ${sim3}`,
+  );
+
+  // Get multiple image embeddings
+  const catImg1 =
+    "https://i.pinimg.com/600x315/21/48/7e/21487e8e0970dd366dafaed6ab25d8d8.jpg";
+  const catImg2 =
+    "https://i.pinimg.com/736x/c9/f2/3e/c9f23e212529f13f19bad5602d84b78b.jpg";
+  const imageEmbeddings = await jina.getImageEmbeddings([catImg1, catImg2]);
+  const sim4 = similarity(
+    imageEmbeddings[0],
+    imageEmbeddings[1],
+    SimilarityType.DEFAULT,
+  );
+  console.log(`Similarity between the two online cat images is ${sim4}`);
+
+  // Get image embeddings from multiple local files
+  const documents = await new SimpleDirectoryReader().loadData({
+    directoryPath: path.join("multimodal", "data"),
+  });
+  const localImages = documents
+    .filter((doc) => doc instanceof ImageDocument)
+    .slice(0, 2); // Get only the first two images
+  const localImageEmbeddings = await jina.getImageEmbeddings(
+    localImages.map((doc) => (doc as ImageDocument).image),
+  );
+  const sim5 = similarity(
+    localImageEmbeddings[0],
+    localImageEmbeddings[1],
+    SimilarityType.DEFAULT,
+  );
+  console.log(`Similarity between the two local images is ${sim5}`);
+}
+
+void main();

examples/package.json

@@ -9,7 +9,7 @@
     "@llamaindex/core": "^0.1.0",
     "@notionhq/client": "^2.2.15",
     "@pinecone-database/pinecone": "^2.2.2",
-    "@zilliz/milvus2-sdk-node": "^2.4.2",
+    "@zilliz/milvus2-sdk-node": "^2.4.4",
     "chromadb": "^1.8.1",
     "commander": "^12.1.0",
     "dotenv": "^16.4.5",
@@ -21,7 +21,7 @@
   "devDependencies": {
     "@types/node": "^20.14.1",
     "tsx": "^4.15.6",
-    "typescript": "^5.5.2"
+    "typescript": "^5.5.3"
   },
   "scripts": {
     "lint": "eslint ."

examples/qdrantdb/preFilters.ts

@@ -64,7 +64,7 @@ async function main() {
           {
             key: "dogId",
             value: "2",
-            filterType: "ExactMatch",
+            operator: "==",
           },
         ],
       },

examples/readers/package.json

@@ -11,8 +11,10 @@
     "start:pdf": "node --import tsx ./src/pdf.ts",
     "start:llamaparse": "node --import tsx ./src/llamaparse.ts",
     "start:notion": "node --import tsx ./src/notion.ts",
+    "start:assemblyai": "node --import tsx ./src/assemblyai.ts",
     "start:llamaparse-dir": "node --import tsx ./src/simple-directory-reader-with-llamaparse.ts",
-    "start:llamaparse-json": "node --import tsx ./src/llamaparse-json.ts"
+    "start:llamaparse-json": "node --import tsx ./src/llamaparse-json.ts",
+    "start:discord": "node --import tsx ./src/discord.ts"
   },
   "dependencies": {
     "llamaindex": "*"
@@ -20,6 +22,6 @@
   "devDependencies": {
     "@types/node": "^20.12.11",
     "tsx": "^4.15.6",
-    "typescript": "^5.5.2"
+    "typescript": "^5.5.3"
   }
 }

examples/readers/src/discord.ts

@@ -0,0 +1,20 @@
+import { DiscordReader } from "llamaindex";
+
+async function main() {
+  // Create an instance of the DiscordReader. Set token here or DISCORD_TOKEN environment variable
+  const discordReader = new DiscordReader();
+
+  // Specify the channel IDs you want to read messages from as an arry of strings
+  const channelIds = ["721374320794009630", "719596376261918720"];
+
+  // Specify the number of messages to fetch per channel
+  const limit = 10;
+
+  // Load messages from the specified channel
+  const messages = await discordReader.loadData(channelIds, limit, true);
+
+  // Print out the messages
+  console.log(messages);
+}
+
+main().catch(console.error);

examples/readers/src/json.ts

@@ -0,0 +1,49 @@
+import { JSONReader } from "llamaindex";
+
+async function main() {
+  // Data
+  const file = "../data/tinytweets.json";
+  const nonAsciiContent = '{"message": "こんにちは世界"}';
+  const jsonlContent = '{"tweet": "Hello world"}\n{"tweet": "こんにちは世界"}';
+
+  // Convert strings to Uint8Array for loadDataAsContent
+  const nonAsciiBuffer = new TextEncoder().encode(nonAsciiContent);
+  const jsonlBuffer = new TextEncoder().encode(jsonlContent);
+
+  // Default settings
+  const reader1 = new JSONReader();
+  const docs1 = await reader1.loadData(file);
+  console.log(docs1[0]);
+
+  // Unclean JSON
+  const reader2 = new JSONReader({ cleanJson: false });
+  const docs2 = await reader2.loadData(file);
+  console.log(docs2[0]);
+
+  // Depth first yield of JSON structural paths, going back 2 levels
+  const reader3 = new JSONReader({ levelsBack: 2 });
+  const docs3 = await reader3.loadData(file);
+  console.log(docs3[0]);
+
+  // Depth first yield of all levels
+  const reader4 = new JSONReader({ levelsBack: 0 });
+  const docs4 = await reader4.loadData(file);
+  console.log(docs4[0]);
+
+  // Depth first yield of all levels, collapse structural paths below length 100
+  const reader5 = new JSONReader({ levelsBack: 0, collapseLength: 100 });
+  const docs5 = await reader5.loadData(file);
+  console.log(docs5[0]);
+
+  // Convert ASCII to unichode escape sequences
+  const reader6 = new JSONReader({ ensureAscii: true });
+  const docs6 = await reader6.loadDataAsContent(nonAsciiBuffer);
+  console.log(docs6[0]);
+
+  // JSON Lines Format
+  const reader7 = new JSONReader({ isJsonLines: true });
+  const docs7 = await reader7.loadDataAsContent(jsonlBuffer);
+  console.log(docs7[0]);
+}
+
+main().catch(console.error);

examples/rerankers/CohereReranker.ts

@@ -15,9 +15,9 @@ async function main() {
 
   const index = await VectorStoreIndex.fromDocuments([document]);
 
-  const retriever = index.asRetriever();
-
-  retriever.similarityTopK = 5;
+  const retriever = index.asRetriever({
+    similarityTopK: 5,
+  });
 
   const nodePostprocessor = new CohereRerank({
     apiKey: "<COHERE_API_KEY>",

examples/split.ts

@@ -8,7 +8,7 @@ async function main() {
 
   const textSplitter = new SentenceSplitter();
 
-  const chunks = textSplitter.splitTextWithOverlaps(essay);
+  const chunks = textSplitter.splitText(essay);
 
   console.log(chunks);
 }

examples/vectorIndexCustomize.ts

@@ -18,8 +18,9 @@ async function main() {
 
   const index = await VectorStoreIndex.fromDocuments([document]);
 
-  const retriever = index.asRetriever();
-  retriever.similarityTopK = 5;
+  const retriever = index.asRetriever({
+    similarityTopK: 5,
+  });
   const nodePostprocessor = new SimilarityPostprocessor({
     similarityCutoff: 0.7,
   });

package.json

@@ -21,19 +21,19 @@
     "@changesets/cli": "^2.27.5",
     "@typescript-eslint/eslint-plugin": "^7.13.1",
     "eslint": "^8.57.0",
-    "eslint-config-next": "^14.2.4",
+    "eslint-config-next": "^14.2.5",
     "eslint-config-prettier": "^9.1.0",
     "eslint-config-turbo": "^2.0.5",
-    "eslint-plugin-react": "7.34.1",
+    "eslint-plugin-react": "7.34.3",
     "husky": "^9.0.11",
     "lint-staged": "^15.2.7",
     "madge": "^7.0.0",
     "prettier": "^3.3.2",
-    "prettier-plugin-organize-imports": "^3.2.4",
+    "prettier-plugin-organize-imports": "^4.0.0",
     "turbo": "^2.0.5",
-    "typescript": "^5.5.2"
+    "typescript": "^5.5.3"
   },
-  "packageManager": "pnpm@9.4.0",
+  "packageManager": "pnpm@9.5.0",
   "pnpm": {
     "overrides": {
       "trim": "1.0.1",

packages/autotool/CHANGELOG.md

@@ -1,5 +1,13 @@
 # @llamaindex/autotool
 
+## 2.0.1
+
+### Patch Changes
+
+- 58abc57: fix: align version
+- Updated dependencies [58abc57]
+  - llamaindex@0.5.16
+
 ## 2.0.0
 
 ### Patch Changes

packages/autotool/examples/01_node/package.json

@@ -5,7 +5,7 @@
   "dependencies": {
     "@llamaindex/autotool": "workspace:*",
     "llamaindex": "workspace:*",
-    "openai": "^4.52.0"
+    "openai": "^4.52.5"
   },
   "devDependencies": {
     "tsx": "^4.15.6"

packages/autotool/examples/02_nextjs/CHANGELOG.md

@@ -1,5 +1,128 @@
 # @llamaindex/autotool-02-next-example
 
+## 0.1.41
+
+### Patch Changes
+
+- Updated dependencies [58abc57]
+  - @llamaindex/autotool@2.0.1
+  - llamaindex@0.5.16
+
+## 0.1.40
+
+### Patch Changes
+
+- Updated dependencies [01c184c]
+- Updated dependencies [07a275f]
+  - llamaindex@0.5.15
+  - @llamaindex/autotool@2.0.0
+
+## 0.1.39
+
+### Patch Changes
+
+- Updated dependencies [c825a2f]
+  - llamaindex@0.5.14
+  - @llamaindex/autotool@2.0.0
+
+## 0.1.38
+
+### Patch Changes
+
+- llamaindex@0.5.13
+- @llamaindex/autotool@2.0.0
+
+## 0.1.37
+
+### Patch Changes
+
+- Updated dependencies [345300f]
+- Updated dependencies [da5cfc4]
+- Updated dependencies [da5cfc4]
+  - llamaindex@0.5.12
+  - @llamaindex/autotool@2.0.0
+
+## 0.1.36
+
+### Patch Changes
+
+- llamaindex@0.5.11
+- @llamaindex/autotool@2.0.0
+
+## 0.1.35
+
+### Patch Changes
+
+- Updated dependencies [086b940]
+- Updated dependencies [5d5716b]
+- Updated dependencies [91d02a4]
+- Updated dependencies [fb6db45]
+  - llamaindex@0.5.10
+  - @llamaindex/autotool@2.0.0
+
+## 0.1.34
+
+### Patch Changes
+
+- Updated dependencies [15962b3]
+  - llamaindex@0.5.9
+  - @llamaindex/autotool@2.0.0
+
+## 0.1.33
+
+### Patch Changes
+
+- Updated dependencies [3d5ba08]
+- Updated dependencies [d917cdc]
+  - llamaindex@0.5.8
+  - @llamaindex/autotool@2.0.0
+
+## 0.1.32
+
+### Patch Changes
+
+- Updated dependencies [ec59acd]
+  - llamaindex@0.5.7
+  - @llamaindex/autotool@2.0.0
+
+## 0.1.31
+
+### Patch Changes
+
+- Updated dependencies [2562244]
+- Updated dependencies [325aa51]
+- Updated dependencies [ab700ea]
+- Updated dependencies [92f0782]
+- Updated dependencies [6cf6ae6]
+- Updated dependencies [b7cfe5b]
+  - llamaindex@0.5.6
+  - @llamaindex/autotool@2.0.0
+
+## 0.1.30
+
+### Patch Changes
+
+- Updated dependencies [b974eea]
+  - llamaindex@0.5.5
+  - @llamaindex/autotool@2.0.0
+
+## 0.1.29
+
+### Patch Changes
+
+- Updated dependencies [1a65ead]
+  - llamaindex@0.5.4
+  - @llamaindex/autotool@2.0.0
+
+## 0.1.28
+
+### Patch Changes
+
+- Updated dependencies [9bbbc67]
+- Updated dependencies [b3681bf]
+  - llamaindex@0.5.3
+  - @llamaindex/autotool@2.0.0
+
 ## 0.1.27
 
 ### Patch Changes

packages/autotool/examples/02_nextjs/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@llamaindex/autotool-02-next-example",
   "private": true,
-  "version": "0.1.27",
+  "version": "0.1.41",
   "scripts": {
     "dev": "next dev",
     "build": "next build",
@@ -14,7 +14,7 @@
     "class-variance-authority": "^0.7.0",
     "dotenv": "^16.3.1",
     "llamaindex": "workspace:*",
-    "lucide-react": "^0.378.0",
+    "lucide-react": "^0.407.0",
     "next": "14.3.0-canary.51",
     "react": "^18.3.1",
     "react-dom": "^18.3.1",
@@ -32,6 +32,6 @@
     "cross-env": "^7.0.3",
     "postcss": "^8.4.32",
     "tailwindcss": "^3.4.4",
-    "typescript": "^5.5.2"
+    "typescript": "^5.5.3"
   }
 }

packages/autotool/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@llamaindex/autotool",
   "type": "module",
-  "version": "2.0.0",
+  "version": "2.0.1",
   "description": "auto transpile your JS function to LLM Agent compatible",
   "files": [
     "dist",
@@ -47,11 +47,11 @@
   "dependencies": {
     "@swc/core": "^1.6.3",
     "jotai": "^2.8.3",
-    "typedoc": "^0.25.13",
+    "typedoc": "^0.26.4",
     "unplugin": "^1.10.1"
   },
   "peerDependencies": {
-    "llamaindex": "^0.5.2",
+    "llamaindex": "^0.5.16",
     "openai": "^4",
     "typescript": "^4"
   },
@@ -70,13 +70,13 @@
     "@swc/types": "^0.1.8",
     "@types/json-schema": "^7.0.15",
     "@types/node": "^20.12.11",
-    "bunchee": "5.3.0-beta.0",
+    "bunchee": "5.3.1",
     "llamaindex": "workspace:*",
-    "next": "14.2.3",
+    "next": "14.2.5",
     "rollup": "^4.18.0",
     "tsx": "^4.15.6",
-    "typescript": "^5.5.2",
-    "vitest": "^1.6.0",
+    "typescript": "^5.5.3",
+    "vitest": "^2.0.2",
     "webpack": "^5.92.1"
   }
 }

packages/cloud/CHANGELOG.md

@@ -1,5 +1,17 @@
 # @llamaindex/cloud
 
+## 0.2.2
+
+### Patch Changes
+
+- 58abc57: fix: align version
+
+## 0.2.1
+
+### Patch Changes
+
+- 1f680d7: chore: bump llamacloud api
+
 ## 0.2.0
 
 ### Minor Changes

packages/cloud/package.json

@@ -1,10 +1,10 @@
 {
   "name": "@llamaindex/cloud",
-  "version": "0.2.0",
+  "version": "0.2.2",
   "type": "module",
   "license": "MIT",
   "scripts": {
-    "generate": "pnpm dlx @hey-api/openapi-ts",
+    "generate": "pnpm dlx @hey-api/openapi-ts@0.49.0",
     "build": "pnpm run generate && bunchee"
   },
   "files": [
@@ -35,6 +35,6 @@
   },
   "devDependencies": {
     "@hey-api/openapi-ts": "^0.48.0",
-    "bunchee": "5.3.0-beta.0"
+    "bunchee": "5.3.1"
   }
 }

packages/community/CHANGELOG.md

@@ -1,5 +1,71 @@
 # @llamaindex/community
 
+## 0.0.29
+
+### Patch Changes
+
+- 58abc57: fix: align version
+- Updated dependencies [58abc57]
+  - @llamaindex/core@0.1.8
+  - @llamaindex/env@0.1.9
+
+## 0.0.28
+
+### Patch Changes
+
+- Updated dependencies [04b2f8e]
+  - @llamaindex/core@0.1.7
+
+## 0.0.27
+
+### Patch Changes
+
+- Updated dependencies [0452af9]
+  - @llamaindex/core@0.1.6
+
+## 0.0.26
+
+### Patch Changes
+
+- 224d507: fix: prevent tool calling getting mixed with conversation
+- 376d29a: feat: added tool calling and agent support for llama3.1 504B
+
+## 0.0.25
+
+### Patch Changes
+
+- Updated dependencies [91d02a4]
+  - @llamaindex/core@0.1.5
+
+## 0.0.24
+
+### Patch Changes
+
+- 3d9a802: feat: added llama 3.1
+- Updated dependencies [15962b3]
+  - @llamaindex/core@0.1.4
+
+## 0.0.23
+
+### Patch Changes
+
+- Updated dependencies [6cf6ae6]
+  - @llamaindex/core@0.1.3
+
+## 0.0.22
+
+### Patch Changes
+
+- Updated dependencies [b974eea]
+  - @llamaindex/core@0.1.2
+
+## 0.0.21
+
+### Patch Changes
+
+- Updated dependencies [b3681bf]
+  - @llamaindex/core@0.1.1
+
 ## 0.0.20
 
 ### Patch Changes

packages/community/README.md

@@ -5,7 +5,8 @@
 ## Current Features:
 
 - Bedrock support for the Anthropic Claude Models [usage](https://ts.llamaindex.ai/modules/llms/available_llms/bedrock)
-- Bedrock support for the Meta LLama 2 and 3 Models [usage](https://ts.llamaindex.ai/modules/llms/available_llms/bedrock)
+- Bedrock support for the Meta LLama 2, 3 and 3.1 Models [usage](https://ts.llamaindex.ai/modules/llms/available_llms/bedrock)
+- Meta LLama3.1 405b tool call support
 
 ## LICENSE
 

packages/community/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@llamaindex/community",
   "description": "Community package for LlamaIndexTS",
-  "version": "0.0.20",
+  "version": "0.0.29",
   "type": "module",
   "types": "dist/type/index.d.ts",
   "main": "dist/cjs/index.js",
@@ -19,11 +19,11 @@
     "./llm/bedrock": {
       "import": {
         "types": "./dist/type/llm/bedrock.d.ts",
-        "default": "./dist/llm/bedrock/base.js"
+        "default": "./dist/llm/bedrock/index.js"
       },
       "require": {
         "types": "./dist/type/llm/bedrock.d.ts",
-        "default": "./dist/llm/bedrock/base.cjs"
+        "default": "./dist/llm/bedrock/index.cjs"
       }
     }
   },
@@ -43,10 +43,11 @@
   },
   "devDependencies": {
     "@types/node": "^20.14.2",
-    "bunchee": "5.3.0-beta.0"
+    "bunchee": "5.3.1"
   },
   "dependencies": {
-    "@aws-sdk/client-bedrock-runtime": "^3.600.0",
-    "@llamaindex/core": "workspace:*"
+    "@aws-sdk/client-bedrock-runtime": "^3.613.0",
+    "@llamaindex/core": "workspace:*",
+    "@llamaindex/env": "workspace:*"
   }
 }

packages/community/src/index.ts

@@ -2,4 +2,4 @@ export {
   BEDROCK_MODELS,
   BEDROCK_MODEL_MAX_TOKENS,
   Bedrock,
-} from "./llm/bedrock/base.js";
+} from "./llm/bedrock/index.js";

packages/community/src/llm/bedrock/anthropic/provider.ts

@@ -16,17 +16,18 @@ import {
   type BedrockChatStreamResponse,
   Provider,
 } from "../provider";
+import { toUtf8 } from "../utils";
 import type {
   AnthropicNoneStreamingResponse,
   AnthropicStreamEvent,
   AnthropicTextContent,
   ToolBlock,
-} from "../types";
+} from "./types";
+
 import {
   mapBaseToolsToAnthropicTools,
   mapChatMessagesToAnthropicMessages,
-  toUtf8,
-} from "../utils";
+} from "./utils";
 
 export class AnthropicProvider extends Provider<AnthropicStreamEvent> {
   getResultFromResponse(
@@ -69,6 +70,7 @@ export class AnthropicProvider extends Provider<AnthropicStreamEvent> {
     let tool: ToolBlock | undefined = undefined;
     // #TODO this should be broken down into a separate consumer
     for await (const response of stream) {
+      const delta = this.getTextFromStreamResponse(response);
       const event = this.getStreamingEventResponse(response);
       if (
         event?.type === "content_block_start" &&
@@ -114,11 +116,10 @@ export class AnthropicProvider extends Provider<AnthropicStreamEvent> {
           };
         }
       }
-      const delta = this.getTextFromStreamResponse(response);
       if (!delta && !options) continue;
 
       yield {
-        delta,
+        delta: options ? "" : delta,
         options,
         raw: response,
       };

packages/community/src/llm/bedrock/anthropic/types.ts

@@ -0,0 +1,142 @@
+import type { ToolMetadata } from "@llamaindex/core/llms";
+import type { InvocationMetrics } from "../types";
+
+type Usage = {
+  input_tokens: number;
+  output_tokens: number;
+};
+
+type Message = {
+  id: string;
+  type: string;
+  role: string;
+  content: string[];
+  model: string;
+  stop_reason: string | null;
+  stop_sequence: string | null;
+  usage: Usage;
+};
+
+export type ToolBlock = {
+  id: string;
+  input: unknown;
+  name: string;
+  type: "tool_use";
+};
+
+export type TextBlock = {
+  type: "text";
+  text: string;
+};
+
+type ContentBlockStart = {
+  type: "content_block_start";
+  index: number;
+  content_block: ToolBlock | TextBlock;
+};
+
+type Delta =
+  | {
+      type: "text_delta";
+      text: string;
+    }
+  | {
+      type: "input_json_delta";
+      partial_json: string;
+    };
+
+type ContentBlockDelta = {
+  type: "content_block_delta";
+  index: number;
+  delta: Delta;
+};
+
+type ContentBlockStop = {
+  type: "content_block_stop";
+  index: number;
+};
+
+type MessageDelta = {
+  type: "message_delta";
+  delta: {
+    stop_reason: string;
+    stop_sequence: string | null;
+  };
+  usage: Usage;
+};
+
+export type MessageStop = {
+  type: "message_stop";
+  "amazon-bedrock-invocationMetrics": InvocationMetrics;
+};
+
+export type AnthropicStreamEvent =
+  | { type: "message_start"; message: Message }
+  | ContentBlockStart
+  | ContentBlockDelta
+  | ContentBlockStop
+  | MessageDelta
+  | MessageStop;
+
+export type AnthropicContent =
+  | AnthropicTextContent
+  | AnthropicImageContent
+  | AnthropicToolContent
+  | AnthropicToolResultContent;
+
+export type AnthropicTextContent = {
+  type: "text";
+  text: string;
+};
+
+export type AnthropicToolContent = {
+  type: "tool_use";
+  id: string;
+  name: string;
+  input: Record<string, unknown>;
+};
+
+export type AnthropicToolResultContent = {
+  type: "tool_result";
+  tool_use_id: string;
+  content: string;
+};
+
+export type AnthropicMediaTypes =
+  | "image/jpeg"
+  | "image/png"
+  | "image/webp"
+  | "image/gif";
+
+export type AnthropicImageSource = {
+  type: "base64";
+  media_type: AnthropicMediaTypes;
+  data: string; // base64 encoded image bytes
+};
+
+export type AnthropicImageContent = {
+  type: "image";
+  source: AnthropicImageSource;
+};
+
+export type AnthropicMessage = {
+  role: "user" | "assistant";
+  content: AnthropicContent[];
+};
+
+export type AnthropicNoneStreamingResponse = {
+  id: string;
+  type: "message";
+  role: "assistant";
+  content: AnthropicContent[];
+  model: string;
+  stop_reason: "end_turn" | "max_tokens" | "stop_sequence";
+  stop_sequence?: string;
+  usage: { input_tokens: number; output_tokens: number };
+};
+
+export type AnthropicTool = {
+  name: string;
+  description: string;
+  input_schema: ToolMetadata["parameters"];
+};

packages/community/src/llm/bedrock/anthropic/utils.ts

@@ -0,0 +1,186 @@
+import type { JSONObject } from "@llamaindex/core/global";
+import type {
+  BaseTool,
+  ChatMessage,
+  MessageContent,
+  MessageContentDetail,
+  ToolCallLLMMessageOptions,
+} from "@llamaindex/core/llms";
+import { mapMessageContentToMessageContentDetails } from "../utils";
+import type {
+  AnthropicContent,
+  AnthropicImageContent,
+  AnthropicMediaTypes,
+  AnthropicMessage,
+  AnthropicTextContent,
+  AnthropicTool,
+} from "./types.js";
+
+const ACCEPTED_IMAGE_MIME_TYPES = [
+  "image/jpeg",
+  "image/png",
+  "image/webp",
+  "image/gif",
+];
+
+export const mergeNeighboringSameRoleMessages = (
+  messages: AnthropicMessage[],
+): AnthropicMessage[] => {
+  return messages.reduce(
+    (result: AnthropicMessage[], current: AnthropicMessage, index: number) => {
+      if (index > 0 && messages[index - 1].role === current.role) {
+        result[result.length - 1].content = [
+          ...result[result.length - 1].content,
+          ...current.content,
+        ];
+      } else {
+        result.push(current);
+      }
+      return result;
+    },
+    [],
+  );
+};
+
+export const mapMessageContentDetailToAnthropicContent = <
+  T extends MessageContentDetail,
+>(
+  detail: T,
+): AnthropicContent => {
+  let content: AnthropicContent;
+
+  if (detail.type === "text") {
+    content = mapTextContent(detail.text);
+  } else if (detail.type === "image_url") {
+    content = mapImageContent(detail.image_url.url);
+  } else {
+    throw new Error("Unsupported content detail type");
+  }
+  return content;
+};
+
+export const mapMessageContentToAnthropicContent = <T extends MessageContent>(
+  content: T,
+): AnthropicContent[] => {
+  return mapMessageContentToMessageContentDetails(content).map(
+    mapMessageContentDetailToAnthropicContent,
+  );
+};
+
+export const mapBaseToolsToAnthropicTools = (
+  tools?: BaseTool[],
+): AnthropicTool[] => {
+  if (!tools) return [];
+  return tools.map((tool: BaseTool) => {
+    const {
+      metadata: { parameters, ...options },
+    } = tool;
+    return {
+      ...options,
+      input_schema: parameters,
+    };
+  });
+};
+
+export const mapChatMessagesToAnthropicMessages = <
+  T extends ChatMessage<ToolCallLLMMessageOptions>,
+>(
+  messages: T[],
+): AnthropicMessage[] => {
+  const mapped = messages
+    .flatMap((msg: T): AnthropicMessage[] => {
+      if (msg.options && "toolCall" in msg.options) {
+        return [
+          {
+            role: "assistant",
+            content: msg.options.toolCall.map((call) => ({
+              type: "tool_use",
+              id: call.id,
+              name: call.name,
+              input: call.input as JSONObject,
+            })),
+          },
+        ];
+      }
+      if (msg.options && "toolResult" in msg.options) {
+        return [
+          {
+            role: "user",
+            content: [
+              {
+                type: "tool_result",
+                tool_use_id: msg.options.toolResult.id,
+                content: msg.options.toolResult.result,
+              },
+            ],
+          },
+        ];
+      }
+      return mapMessageContentToMessageContentDetails(msg.content).map(
+        (detail: MessageContentDetail): AnthropicMessage => {
+          const content = mapMessageContentDetailToAnthropicContent(detail);
+
+          return {
+            role: msg.role === "assistant" ? "assistant" : "user",
+            content: [content],
+          };
+        },
+      );
+    })
+    .filter((message: AnthropicMessage) => {
+      const content = message.content[0];
+      if (content.type === "text" && !content.text) return false;
+      if (content.type === "image" && !content.source.data) return false;
+      if (content.type === "image" && message.role === "assistant")
+        return false;
+      return true;
+    });
+
+  return mergeNeighboringSameRoleMessages(mapped);
+};
+
+export const mapTextContent = (text: string): AnthropicTextContent => {
+  return { type: "text", text };
+};
+
+export const extractDataUrlComponents = (
+  dataUrl: string,
+): {
+  mimeType: string;
+  base64: string;
+} => {
+  const parts = dataUrl.split(";base64,");
+
+  if (parts.length !== 2 || !parts[0].startsWith("data:")) {
+    throw new Error("Invalid data URL");
+  }
+
+  const mimeType = parts[0].slice(5);
+  const base64 = parts[1];
+
+  return {
+    mimeType,
+    base64,
+  };
+};
+
+export const mapImageContent = (imageUrl: string): AnthropicImageContent => {
+  if (!imageUrl.startsWith("data:"))
+    throw new Error(
+      "For Anthropic please only use base64 data url, e.g.: data:image/jpeg;base64,SGVsbG8sIFdvcmxkIQ==",
+    );
+  const { mimeType, base64: data } = extractDataUrlComponents(imageUrl);
+  if (!ACCEPTED_IMAGE_MIME_TYPES.includes(mimeType))
+    throw new Error(
+      `Anthropic only accepts the following mimeTypes: ${ACCEPTED_IMAGE_MIME_TYPES.join("\n")}`,
+    );
+
+  return {
+    type: "image",
+    source: {
+      type: "base64",
+      media_type: mimeType as AnthropicMediaTypes,
+      data,
+    },
+  };
+};

packages/community/src/llm/bedrock/index.ts

@@ -22,8 +22,16 @@ import {
   type BedrockChatStreamResponse,
   Provider,
 } from "./provider";
-import { PROVIDERS } from "./providers";
-import { mapMessageContentToMessageContentDetails } from "./utils.js";
+import { mapMessageContentToMessageContentDetails } from "./utils";
+
+import { AnthropicProvider } from "./anthropic/provider";
+import { MetaProvider } from "./meta/provider";
+
+// Other providers should go here
+export const PROVIDERS: { [key: string]: Provider } = {
+  anthropic: new AnthropicProvider(),
+  meta: new MetaProvider(),
+};
 
 export type BedrockChatParamsStreaming = LLMChatParamsStreaming<
   BedrockAdditionalChatOptions,
@@ -60,6 +68,9 @@ export enum BEDROCK_MODELS {
   META_LLAMA2_70B_CHAT = "meta.llama2-70b-chat-v1",
   META_LLAMA3_8B_INSTRUCT = "meta.llama3-8b-instruct-v1:0",
   META_LLAMA3_70B_INSTRUCT = "meta.llama3-70b-instruct-v1:0",
+  META_LLAMA3_1_8B_INSTRUCT = "meta.llama3-1-8b-instruct-v1:0",
+  META_LLAMA3_1_70B_INSTRUCT = "meta.llama3-1-70b-instruct-v1:0",
+  META_LLAMA3_1_405B_INSTRUCT = "meta.llama3-1-405b-instruct-v1:0",
   MISTRAL_7B_INSTRUCT = "mistral.mistral-7b-instruct-v0:2",
   MISTRAL_MIXTRAL_7B_INSTRUCT = "mistral.mixtral-8x7b-instruct-v0:1",
   MISTRAL_MIXTRAL_LARGE_2402 = "mistral.mistral-large-2402-v1:0",
@@ -94,6 +105,9 @@ const CHAT_ONLY_MODELS = {
   [BEDROCK_MODELS.META_LLAMA2_70B_CHAT]: 4096,
   [BEDROCK_MODELS.META_LLAMA3_8B_INSTRUCT]: 8192,
   [BEDROCK_MODELS.META_LLAMA3_70B_INSTRUCT]: 8192,
+  [BEDROCK_MODELS.META_LLAMA3_1_8B_INSTRUCT]: 128000,
+  [BEDROCK_MODELS.META_LLAMA3_1_70B_INSTRUCT]: 128000,
+  [BEDROCK_MODELS.META_LLAMA3_1_405B_INSTRUCT]: 128000,
   [BEDROCK_MODELS.MISTRAL_7B_INSTRUCT]: 32000,
   [BEDROCK_MODELS.MISTRAL_MIXTRAL_7B_INSTRUCT]: 32000,
   [BEDROCK_MODELS.MISTRAL_MIXTRAL_LARGE_2402]: 32000,
@@ -121,6 +135,9 @@ export const STREAMING_MODELS = new Set([
   BEDROCK_MODELS.META_LLAMA2_70B_CHAT,
   BEDROCK_MODELS.META_LLAMA3_8B_INSTRUCT,
   BEDROCK_MODELS.META_LLAMA3_70B_INSTRUCT,
+  BEDROCK_MODELS.META_LLAMA3_1_8B_INSTRUCT,
+  BEDROCK_MODELS.META_LLAMA3_1_70B_INSTRUCT,
+  BEDROCK_MODELS.META_LLAMA3_1_405B_INSTRUCT,
   BEDROCK_MODELS.MISTRAL_7B_INSTRUCT,
   BEDROCK_MODELS.MISTRAL_MIXTRAL_7B_INSTRUCT,
   BEDROCK_MODELS.MISTRAL_MIXTRAL_LARGE_2402,
@@ -131,6 +148,7 @@ export const TOOL_CALL_MODELS = [
   BEDROCK_MODELS.ANTHROPIC_CLAUDE_3_HAIKU,
   BEDROCK_MODELS.ANTHROPIC_CLAUDE_3_OPUS,
   BEDROCK_MODELS.ANTHROPIC_CLAUDE_3_5_SONNET,
+  BEDROCK_MODELS.META_LLAMA3_1_405B_INSTRUCT,
 ];
 
 const getProvider = (model: string): Provider => {
@@ -160,6 +178,9 @@ export const BEDROCK_MODEL_MAX_TOKENS: Partial<Record<BEDROCK_MODELS, number>> =
     [BEDROCK_MODELS.META_LLAMA2_70B_CHAT]: 2048,
     [BEDROCK_MODELS.META_LLAMA3_8B_INSTRUCT]: 2048,
     [BEDROCK_MODELS.META_LLAMA3_70B_INSTRUCT]: 2048,
+    [BEDROCK_MODELS.META_LLAMA3_1_8B_INSTRUCT]: 2048,
+    [BEDROCK_MODELS.META_LLAMA3_1_70B_INSTRUCT]: 2048,
+    [BEDROCK_MODELS.META_LLAMA3_1_405B_INSTRUCT]: 2048,
   };
 
 const DEFAULT_BEDROCK_PARAMS = {

packages/community/src/llm/bedrock/meta/constants.ts

@@ -0,0 +1,3 @@
+export const TOKENS = {
+  TOOL_CALL: "<|python_tag|>",
+};

packages/community/src/llm/bedrock/meta/provider.ts

@@ -0,0 +1,136 @@
+import type {
+  InvokeModelCommandInput,
+  InvokeModelWithResponseStreamCommandInput,
+  ResponseStream,
+} from "@aws-sdk/client-bedrock-runtime";
+import type {
+  BaseTool,
+  ChatMessage,
+  LLMMetadata,
+  ToolCall,
+  ToolCallLLMMessageOptions,
+} from "@llamaindex/core/llms";
+import { toUtf8 } from "../utils";
+import type { MetaNoneStreamingResponse, MetaStreamEvent } from "./types";
+
+import { randomUUID } from "@llamaindex/env";
+import { Provider, type BedrockChatStreamResponse } from "../provider";
+import { TOKENS } from "./constants";
+import {
+  mapChatMessagesToMetaLlama2Messages,
+  mapChatMessagesToMetaLlama3Messages,
+} from "./utils";
+
+export class MetaProvider extends Provider<MetaStreamEvent> {
+  getResultFromResponse(
+    response: Record<string, any>,
+  ): MetaNoneStreamingResponse {
+    return JSON.parse(toUtf8(response.body));
+  }
+
+  getToolsFromResponse<ToolContent>(
+    response: Record<string, any>,
+  ): ToolContent[] {
+    const result = this.getResultFromResponse(response);
+    if (!result.generation.trim().startsWith(TOKENS.TOOL_CALL)) return [];
+    const tool = JSON.parse(
+      result.generation.trim().split(TOKENS.TOOL_CALL)[1],
+    );
+    return [
+      {
+        id: randomUUID(),
+        name: tool.name,
+        input: tool.parameters,
+      } as ToolContent,
+    ];
+  }
+
+  getTextFromResponse(response: Record<string, any>): string {
+    const result = this.getResultFromResponse(response);
+    if (result.generation.trim().startsWith(TOKENS.TOOL_CALL)) return "";
+    return result.generation;
+  }
+
+  getTextFromStreamResponse(response: Record<string, any>): string {
+    const event = this.getStreamingEventResponse(response);
+    if (event?.generation) {
+      return event.generation;
+    }
+    return "";
+  }
+
+  async *reduceStream(
+    stream: AsyncIterable<ResponseStream>,
+  ): BedrockChatStreamResponse {
+    const collecting: string[] = [];
+    let toolId: string | undefined = undefined;
+    for await (const response of stream) {
+      const event = this.getStreamingEventResponse(response);
+      const delta = this.getTextFromStreamResponse(response);
+      // odd quirk of llama3.1, start token is \n\n
+      if (
+        !event?.generation.trim() &&
+        event?.generation_token_count === 1 &&
+        event.prompt_token_count !== null
+      )
+        continue;
+
+      if (delta === TOKENS.TOOL_CALL) {
+        toolId = randomUUID();
+        continue;
+      }
+
+      let options: undefined | ToolCallLLMMessageOptions = undefined;
+      if (toolId && event?.stop_reason === "stop") {
+        const tool = JSON.parse(collecting.join(""));
+        options = {
+          toolCall: [
+            {
+              id: toolId,
+              name: tool.name,
+              input: tool.parameters,
+            } as ToolCall,
+          ],
+        };
+      } else if (toolId && !event?.stop_reason) {
+        collecting.push(delta);
+        continue;
+      }
+
+      if (!delta && !options) continue;
+
+      yield {
+        delta: options ? "" : delta,
+        options,
+        raw: response,
+      };
+    }
+  }
+
+  getRequestBody<T extends ChatMessage>(
+    metadata: LLMMetadata,
+    messages: T[],
+    tools?: BaseTool[],
+  ): InvokeModelCommandInput | InvokeModelWithResponseStreamCommandInput {
+    let prompt: string = "";
+    if (metadata.model.startsWith("meta.llama3")) {
+      prompt = mapChatMessagesToMetaLlama3Messages(messages, tools);
+    } else if (metadata.model.startsWith("meta.llama2")) {
+      prompt = mapChatMessagesToMetaLlama2Messages(messages);
+    } else {
+      throw new Error(`Meta model ${metadata.model} is not supported`);
+    }
+
+    return {
+      modelId: metadata.model,
+      contentType: "application/json",
+      accept: "application/json",
+      body: JSON.stringify({
+        prompt,
+        max_gen_len: metadata.maxTokens,
+        temperature: metadata.temperature,
+        top_p: metadata.topP,
+      }),
+    };
+  }
+}

packages/community/src/llm/bedrock/meta/types.ts

@@ -0,0 +1,21 @@
+import type { InvocationMetrics } from "../types";
+
+export type MetaTextContent = string;
+
+export type MetaMessage = {
+  role: "user" | "assistant" | "system" | "ipython";
+  content: MetaTextContent;
+};
+
+type MetaResponse = {
+  generation: string;
+  prompt_token_count: number;
+  generation_token_count: number;
+  stop_reason: "stop" | "length";
+};
+
+export type MetaStreamEvent = MetaResponse & {
+  "amazon-bedrock-invocationMetrics": InvocationMetrics;
+};
+
+export type MetaNoneStreamingResponse = MetaResponse;

packages/community/src/llm/bedrock/meta/utils.ts

@@ -0,0 +1,198 @@
+import type {
+  BaseTool,
+  ChatMessage,
+  MessageContentTextDetail,
+  ToolCallLLMMessageOptions,
+} from "@llamaindex/core/llms";
+import type { MetaMessage } from "./types";
+
+const getToolCallInstructionString = (tool: BaseTool): string => {
+  return `Use the function '${tool.metadata.name}' to '${tool.metadata.description}'`;
+};
+
+const getToolCallParametersString = (tool: BaseTool): string => {
+  return JSON.stringify({
+    name: tool.metadata.name,
+    description: tool.metadata.description,
+    parameters: tool.metadata.parameters
+      ? Object.entries(tool.metadata.parameters.properties).map(
+          ([name, definition]) => ({ [name]: definition }),
+        )
+      : {},
+  });
+};
+
+// ported from https://github.com/meta-llama/llama-agentic-system/blob/main/llama_agentic_system/system_prompt.py
+// NOTE: using json instead of the above xml style tool calling works more reliability
+export const getToolsPrompt = (tools?: BaseTool[]) => {
+  if (!tools?.length) return "";
+
+  const customToolParams = tools.map((tool) => {
+    return [
+      getToolCallInstructionString(tool),
+      getToolCallParametersString(tool),
+    ].join("\n\n");
+  });
+
+  return `
+Environment: node
+
+# Tool Instructions
+- Never use ipython, always use javascript in node
+
+Cutting Knowledge Date: December 2023
+Today Date: ${new Date().toLocaleString("en-US", { year: "numeric", month: "long" })} 
+
+You have access to the following functions:
+
+${customToolParams}
+
+Think very carefully before calling functions.
+
+If a you choose to call a function ONLY reply in the following json format:
+{
+  "name": function_name,
+  "parameters": parameters,
+}
+where
+
+{
+  "name": function_name,
+  "parameters": parameters, => a JSON dict with the function argument name as key and function argument value as value.
+}
+
+Here is an example,
+
+{
+  "name": "example_function_name",
+  "parameters": {"example_name": "example_value"}
+}
+
+Reminder:
+- Function calls MUST follow the specified format
+- Required parameters MUST be specified
+- Only call one function at a time
+- Put the entire function call reply on one line
+- Always add your sources when using search results to answer the user query
+  `;
+};
+
+export const mapChatRoleToMetaRole = (
+  role: ChatMessage["role"],
+): MetaMessage["role"] => {
+  if (role === "assistant") return "assistant";
+  if (role === "user") return "user";
+  return "system";
+};
+
+export const mapChatMessagesToMetaMessages = <
+  T extends ChatMessage<ToolCallLLMMessageOptions>,
+>(
+  messages: T[],
+): MetaMessage[] => {
+  return messages.flatMap((msg) => {
+    if (msg.options && "toolCall" in msg.options) {
+      return msg.options.toolCall.map((call) => ({
+        role: "assistant",
+        content: JSON.stringify({
+          id: call.id,
+          name: call.name,
+          parameters: call.input,
+        }),
+      }));
+    }
+
+    if (msg.options && "toolResult" in msg.options) {
+      return {
+        role: "ipython",
+        content: JSON.stringify(msg.options.toolResult),
+      };
+    }
+
+    let content: string = "";
+    if (typeof msg.content === "string") {
+      content = msg.content;
+    } else if (msg.content.length) {
+      content = (msg.content[0] as MessageContentTextDetail).text;
+    }
+    return {
+      role: mapChatRoleToMetaRole(msg.role),
+      content,
+    };
+  });
+};
+
+/**
+ * Documentation at https://llama.meta.com/docs/model-cards-and-prompt-formats/meta-llama-3
+ */
+export const mapChatMessagesToMetaLlama3Messages = <T extends ChatMessage>(
+  messages: T[],
+  tools?: BaseTool[],
+): string => {
+  const parts: string[] = [];
+  if (tools?.length) {
+    parts.push(
+      "<|begin_of_text|>",
+      "<|start_header_id|>system<|end_header_id|>",
+      getToolsPrompt(tools),
+      "<|eot_id|>",
+    );
+  }
+
+  const mapped = mapChatMessagesToMetaMessages(messages).map((message) => {
+    return [
+      "<|start_header_id|>",
+      message.role,
+      "<|end_header_id|>",
+      message.content,
+      "<|eot_id|>",
+    ].join("\n");
+  });
+
+  parts.push(
+    "<|begin_of_text|>",
+    ...mapped,
+    "<|start_header_id|>assistant<|end_header_id|>",
+  );
+  return parts.join("\n");
+};
+
+/**
+ * Documentation at https://llama.meta.com/docs/model-cards-and-prompt-formats/meta-llama-2
+ */
+export const mapChatMessagesToMetaLlama2Messages = <T extends ChatMessage>(
+  messages: T[],
+): string => {
+  const mapped = mapChatMessagesToMetaMessages(messages);
+  let output = "<s>";
+  let insideInst = false;
+  let needsStartAgain = false;
+  for (const message of mapped) {
+    if (needsStartAgain) {
+      output += "<s>";
+      needsStartAgain = false;
+    }
+    const text = message.content;
+    if (message.role === "system") {
+      if (!insideInst) {
+        output += "[INST] ";
+        insideInst = true;
+      }
+      output += `<<SYS>>\n${text}\n<</SYS>>\n`;
+    } else if (message.role === "user") {
+      output += text;
+      if (insideInst) {
+        output += " [/INST]";
+        insideInst = false;
+      }
+    } else if (message.role === "assistant") {
+      if (insideInst) {
+        output += " [/INST]";
+        insideInst = false;
+      }
+      output += ` ${text} </s>\n`;
+      needsStartAgain = true;
+    }
+  }
+  return output;
+};

packages/community/src/llm/bedrock/provider.ts

@@ -23,6 +23,7 @@ export type BedrockChatStreamResponse = AsyncIterable<
 export abstract class Provider<ProviderStreamEvent extends {} = {}> {
   abstract getTextFromResponse(response: Record<string, any>): string;
 
+  // Return tool calls from none streaming calls
   abstract getToolsFromResponse<T extends {} = {}>(
     response: Record<string, any>,
   ): T[];

packages/community/src/llm/bedrock/providers/index.ts

@@ -1,9 +0,0 @@
-import { Provider } from "../provider";
-import { AnthropicProvider } from "./anthropic";
-import { MetaProvider } from "./meta";
-
-// Other providers should go here
-export const PROVIDERS: { [key: string]: Provider } = {
-  anthropic: new AnthropicProvider(),
-  meta: new MetaProvider(),
-};

packages/community/src/llm/bedrock/providers/meta.ts

@@ -1,69 +0,0 @@
-import type {
-  InvokeModelCommandInput,
-  InvokeModelWithResponseStreamCommandInput,
-} from "@aws-sdk/client-bedrock-runtime";
-import type { ChatMessage, LLMMetadata } from "@llamaindex/core/llms";
-import type { MetaNoneStreamingResponse, MetaStreamEvent } from "../types";
-import {
-  mapChatMessagesToMetaLlama2Messages,
-  mapChatMessagesToMetaLlama3Messages,
-  toUtf8,
-} from "../utils";
-
-import { Provider } from "../provider";
-
-export class MetaProvider extends Provider<MetaStreamEvent> {
-  constructor() {
-    super();
-  }
-
-  getResultFromResponse(
-    response: Record<string, any>,
-  ): MetaNoneStreamingResponse {
-    return JSON.parse(toUtf8(response.body));
-  }
-
-  getToolsFromResponse(_response: Record<string, any>): never {
-    throw new Error("Not supported by this provider.");
-  }
-
-  getTextFromResponse(response: Record<string, any>): string {
-    const result = this.getResultFromResponse(response);
-    return result.generation;
-  }
-
-  getTextFromStreamResponse(response: Record<string, any>): string {
-    const event = this.getStreamingEventResponse(response);
-    if (event?.generation) {
-      return event.generation;
-    }
-    return "";
-  }
-
-  getRequestBody<T extends ChatMessage>(
-    metadata: LLMMetadata,
-    messages: T[],
-  ): InvokeModelCommandInput | InvokeModelWithResponseStreamCommandInput {
-    let promptFunction: (messages: ChatMessage[]) => string;
-
-    if (metadata.model.startsWith("meta.llama3")) {
-      promptFunction = mapChatMessagesToMetaLlama3Messages;
-    } else if (metadata.model.startsWith("meta.llama2")) {
-      promptFunction = mapChatMessagesToMetaLlama2Messages;
-    } else {
-      throw new Error(`Meta model ${metadata.model} is not supported`);
-    }
-
-    return {
-      modelId: metadata.model,
-      contentType: "application/json",
-      accept: "application/json",
-      body: JSON.stringify({
-        prompt: promptFunction(messages),
-        max_gen_len: metadata.maxTokens,
-        temperature: metadata.temperature,
-        top_p: metadata.topP,
-      }),
-    };
-  }
-}

packages/community/src/llm/bedrock/types.ts

@@ -1,165 +1,11 @@
-type Usage = {
-  input_tokens: number;
-  output_tokens: number;
-};
-
-type Message = {
-  id: string;
-  type: string;
-  role: string;
-  content: string[];
-  model: string;
-  stop_reason: string | null;
-  stop_sequence: string | null;
-  usage: Usage;
-};
-
-export type ToolBlock = {
-  id: string;
-  input: unknown;
-  name: string;
-  type: "tool_use";
-};
-
-export type TextBlock = {
-  type: "text";
-  text: string;
-};
-
-type ContentBlockStart = {
-  type: "content_block_start";
-  index: number;
-  content_block: ToolBlock | TextBlock;
-};
-
-type Delta =
-  | {
-      type: "text_delta";
-      text: string;
-    }
-  | {
-      type: "input_json_delta";
-      partial_json: string;
-    };
-
-type ContentBlockDelta = {
-  type: "content_block_delta";
-  index: number;
-  delta: Delta;
-};
-
-type ContentBlockStop = {
-  type: "content_block_stop";
-  index: number;
-};
-
-type MessageDelta = {
-  type: "message_delta";
-  delta: {
-    stop_reason: string;
-    stop_sequence: string | null;
-  };
-  usage: Usage;
-};
-
-type InvocationMetrics = {
+export type InvocationMetrics = {
   inputTokenCount: number;
   outputTokenCount: number;
   invocationLatency: number;
   firstByteLatency: number;
 };
 
-type MessageStop = {
-  type: "message_stop";
-  "amazon-bedrock-invocationMetrics": InvocationMetrics;
-};
-
 export type ToolChoice =
   | { type: "any" }
   | { type: "auto" }
   | { type: "tool"; name: string };
-
-export type AnthropicStreamEvent =
-  | { type: "message_start"; message: Message }
-  | ContentBlockStart
-  | ContentBlockDelta
-  | ContentBlockStop
-  | MessageDelta
-  | MessageStop;
-
-export type AnthropicContent =
-  | AnthropicTextContent
-  | AnthropicImageContent
-  | AnthropicToolContent
-  | AnthropicToolResultContent;
-
-export type MetaTextContent = string;
-
-export type AnthropicTextContent = {
-  type: "text";
-  text: string;
-};
-
-export type AnthropicToolContent = {
-  type: "tool_use";
-  id: string;
-  name: string;
-  input: Record<string, unknown>;
-};
-
-export type AnthropicToolResultContent = {
-  type: "tool_result";
-  tool_use_id: string;
-  content: string;
-};
-
-export type AnthropicMediaTypes =
-  | "image/jpeg"
-  | "image/png"
-  | "image/webp"
-  | "image/gif";
-
-export type AnthropicImageSource = {
-  type: "base64";
-  media_type: AnthropicMediaTypes;
-  data: string; // base64 encoded image bytes
-};
-
-export type AnthropicImageContent = {
-  type: "image";
-  source: AnthropicImageSource;
-};
-
-export type AnthropicMessage = {
-  role: "user" | "assistant";
-  content: AnthropicContent[];
-};
-
-export type MetaMessage = {
-  role: "user" | "assistant" | "system";
-  content: MetaTextContent;
-};
-
-export type AnthropicNoneStreamingResponse = {
-  id: string;
-  type: "message";
-  role: "assistant";
-  content: AnthropicContent[];
-  model: string;
-  stop_reason: "end_turn" | "max_tokens" | "stop_sequence";
-  stop_sequence?: string;
-  usage: { input_tokens: number; output_tokens: number };
-};
-
-type MetaResponse = {
-  generation: string;
-  prompt_token_count: number;
-  generation_token_count: number;
-  stop_reason: "stop" | "length";
-};
-
-export type MetaStreamEvent = MetaResponse & {
-  "amazon-bedrock-invocationMetrics": InvocationMetrics;
-};
-
-export type MetaNoneStreamingResponse = MetaResponse;

packages/community/src/llm/bedrock/utils.ts

@@ -1,28 +1,7 @@
-import type { JSONObject } from "@llamaindex/core/global";
 import type {
-  BaseTool,
-  ChatMessage,
   MessageContent,
   MessageContentDetail,
-  MessageContentTextDetail,
-  ToolCallLLMMessageOptions,
-  ToolMetadata,
 } from "@llamaindex/core/llms";
-import type {
-  AnthropicContent,
-  AnthropicImageContent,
-  AnthropicMediaTypes,
-  AnthropicMessage,
-  AnthropicTextContent,
-  MetaMessage,
-} from "./types.js";
-
-const ACCEPTED_IMAGE_MIME_TYPES = [
-  "image/jpeg",
-  "image/png",
-  "image/webp",
-  "image/gif",
-];
 
 export const mapMessageContentToMessageContentDetails = (
   content: MessageContent,
@@ -30,250 +9,5 @@ export const mapMessageContentToMessageContentDetails = (
   return Array.isArray(content) ? content : [{ type: "text", text: content }];
 };
 
-export const mergeNeighboringSameRoleMessages = (
-  messages: AnthropicMessage[],
-): AnthropicMessage[] => {
-  return messages.reduce(
-    (result: AnthropicMessage[], current: AnthropicMessage, index: number) => {
-      if (index > 0 && messages[index - 1].role === current.role) {
-        result[result.length - 1].content = [
-          ...result[result.length - 1].content,
-          ...current.content,
-        ];
-      } else {
-        result.push(current);
-      }
-      return result;
-    },
-    [],
-  );
-};
-
-export const mapMessageContentDetailToAnthropicContent = <
-  T extends MessageContentDetail,
->(
-  detail: T,
-): AnthropicContent => {
-  let content: AnthropicContent;
-
-  if (detail.type === "text") {
-    content = mapTextContent(detail.text);
-  } else if (detail.type === "image_url") {
-    content = mapImageContent(detail.image_url.url);
-  } else {
-    throw new Error("Unsupported content detail type");
-  }
-  return content;
-};
-
-export const mapMessageContentToAnthropicContent = <T extends MessageContent>(
-  content: T,
-): AnthropicContent[] => {
-  return mapMessageContentToMessageContentDetails(content).map(
-    mapMessageContentDetailToAnthropicContent,
-  );
-};
-
-type AnthropicTool = {
-  name: string;
-  description: string;
-  input_schema: ToolMetadata["parameters"];
-};
-
-export const mapBaseToolsToAnthropicTools = (
-  tools?: BaseTool[],
-): AnthropicTool[] => {
-  if (!tools) return [];
-  return tools.map((tool: BaseTool) => {
-    const {
-      metadata: { parameters, ...options },
-    } = tool;
-    return {
-      ...options,
-      input_schema: parameters,
-    };
-  });
-};
-
-export const mapChatMessagesToAnthropicMessages = <
-  T extends ChatMessage<ToolCallLLMMessageOptions>,
->(
-  messages: T[],
-): AnthropicMessage[] => {
-  const mapped = messages
-    .flatMap((msg: T): AnthropicMessage[] => {
-      if (msg.options && "toolCall" in msg.options) {
-        return [
-          {
-            role: "assistant",
-            content: msg.options.toolCall.map((call) => ({
-              type: "tool_use",
-              id: call.id,
-              name: call.name,
-              input: call.input as JSONObject,
-            })),
-          },
-        ];
-      }
-      if (msg.options && "toolResult" in msg.options) {
-        return [
-          {
-            role: "user",
-            content: [
-              {
-                type: "tool_result",
-                tool_use_id: msg.options.toolResult.id,
-                content: msg.options.toolResult.result,
-              },
-            ],
-          },
-        ];
-      }
-      return mapMessageContentToMessageContentDetails(msg.content).map(
-        (detail: MessageContentDetail): AnthropicMessage => {
-          const content = mapMessageContentDetailToAnthropicContent(detail);
-
-          return {
-            role: msg.role === "assistant" ? "assistant" : "user",
-            content: [content],
-          };
-        },
-      );
-    })
-    .filter((message: AnthropicMessage) => {
-      const content = message.content[0];
-      if (content.type === "text" && !content.text) return false;
-      if (content.type === "image" && !content.source.data) return false;
-      return true;
-    });
-
-  return mergeNeighboringSameRoleMessages(mapped);
-};
-
-export const mapChatMessagesToMetaMessages = <T extends ChatMessage>(
-  messages: T[],
-): MetaMessage[] => {
-  return messages.map((msg) => {
-    let content: string = "";
-    if (typeof msg.content === "string") {
-      content = msg.content;
-    } else if (msg.content.length) {
-      content = (msg.content[0] as MessageContentTextDetail).text;
-    }
-    return {
-      role:
-        msg.role === "assistant"
-          ? "assistant"
-          : msg.role === "user"
-            ? "user"
-            : "system",
-      content,
-    };
-  });
-};
-
-/**
- * Documentation at https://llama.meta.com/docs/model-cards-and-prompt-formats/meta-llama-3
- */
-export const mapChatMessagesToMetaLlama3Messages = <T extends ChatMessage>(
-  messages: T[],
-): string => {
-  const mapped = mapChatMessagesToMetaMessages(messages).map((message) => {
-    const text = message.content;
-    return `<|start_header_id|>${message.role}<|end_header_id|>\n${text}\n<|eot_id|>\n`;
-  });
-  return (
-    "<|begin_of_text|>" +
-    mapped.join("\n") +
-    "\n<|start_header_id|>assistant<|end_header_id|>\n"
-  );
-};
-
-/**
- * Documentation at https://llama.meta.com/docs/model-cards-and-prompt-formats/meta-llama-2
- */
-export const mapChatMessagesToMetaLlama2Messages = <T extends ChatMessage>(
-  messages: T[],
-): string => {
-  const mapped = mapChatMessagesToMetaMessages(messages);
-  let output = "<s>";
-  let insideInst = false;
-  let needsStartAgain = false;
-  for (const message of mapped) {
-    if (needsStartAgain) {
-      output += "<s>";
-      needsStartAgain = false;
-    }
-    const text = message.content;
-    if (message.role === "system") {
-      if (!insideInst) {
-        output += "[INST] ";
-        insideInst = true;
-      }
-      output += `<<SYS>>\n${text}\n<</SYS>>\n`;
-    } else if (message.role === "user") {
-      output += text;
-      if (insideInst) {
-        output += " [/INST]";
-        insideInst = false;
-      }
-    } else if (message.role === "assistant") {
-      if (insideInst) {
-        output += " [/INST]";
-        insideInst = false;
-      }
-      output += ` ${text} </s>\n`;
-      needsStartAgain = true;
-    }
-  }
-  return output;
-};
-
-export const mapTextContent = (text: string): AnthropicTextContent => {
-  return { type: "text", text };
-};
-
-export const extractDataUrlComponents = (
-  dataUrl: string,
-): {
-  mimeType: string;
-  base64: string;
-} => {
-  const parts = dataUrl.split(";base64,");
-
-  if (parts.length !== 2 || !parts[0].startsWith("data:")) {
-    throw new Error("Invalid data URL");
-  }
-
-  const mimeType = parts[0].slice(5);
-  const base64 = parts[1];
-
-  return {
-    mimeType,
-    base64,
-  };
-};
-
-export const mapImageContent = (imageUrl: string): AnthropicImageContent => {
-  if (!imageUrl.startsWith("data:"))
-    throw new Error(
-      "For Anthropic please only use base64 data url, e.g.: data:image/jpeg;base64,SGVsbG8sIFdvcmxkIQ==",
-    );
-  const { mimeType, base64: data } = extractDataUrlComponents(imageUrl);
-  if (!ACCEPTED_IMAGE_MIME_TYPES.includes(mimeType))
-    throw new Error(
-      `Anthropic only accepts the following mimeTypes: ${ACCEPTED_IMAGE_MIME_TYPES.join("\n")}`,
-    );
-
-  return {
-    type: "image",
-    source: {
-      type: "base64",
-      media_type: mimeType as AnthropicMediaTypes,
-      data,
-    },
-  };
-};
-
 export const toUtf8 = (input: Uint8Array): string =>
   new TextDecoder("utf-8").decode(input);

packages/core/CHANGELOG.md

@@ -1,5 +1,60 @@
 # @llamaindex/core
 
+## 0.1.8
+
+### Patch Changes
+
+- 58abc57: fix: align version
+- Updated dependencies [58abc57]
+  - @llamaindex/env@0.1.9
+
+## 0.1.7
+
+### Patch Changes
+
+- 04b2f8e: Fix issue with metadata included after sentence splitter
+
+## 0.1.6
+
+### Patch Changes
+
+- 0452af9: fix: handling errors in splitBySentenceTokenizer
+
+## 0.1.5
+
+### Patch Changes
+
+- 91d02a4: feat: support transform component callable
+
+## 0.1.4
+
+### Patch Changes
+
+- 15962b3: feat: node parser refactor
+
+  Align the text splitter logic with Python; it has almost the same logic as Python; Zod checks for input and better error messages and event system.
+
+  This change will not be considered a breaking change since it doesn't have a significant output difference from the last version,
+  but some edge cases will change, like the page separator and parameter for the constructor.
+
+## 0.1.3
+
+### Patch Changes
+
+- 6cf6ae6: feat: abstract query type
+
+## 0.1.2
+
+### Patch Changes
+
+- b974eea: Add support for Metadata filters
+
+## 0.1.1
+
+### Patch Changes
+
+- b3681bf: fix: DataCloneError when using FunctionTool
+
 ## 0.1.0
 
 ### Minor Changes

packages/core/package.json

@@ -1,9 +1,37 @@
 {
   "name": "@llamaindex/core",
   "type": "module",
-  "version": "0.1.0",
+  "version": "0.1.8",
   "description": "LlamaIndex Core Module",
   "exports": {
+    "./node-parser": {
+      "require": {
+        "types": "./dist/node-parser/index.d.cts",
+        "default": "./dist/node-parser/index.cjs"
+      },
+      "import": {
+        "types": "./dist/node-parser/index.d.ts",
+        "default": "./dist/node-parser/index.js"
+      },
+      "default": {
+        "types": "./dist/node-parser/index.d.ts",
+        "default": "./dist/node-parser/index.js"
+      }
+    },
+    "./query-engine": {
+      "require": {
+        "types": "./dist/query-engine/index.d.cts",
+        "default": "./dist/query-engine/index.cjs"
+      },
+      "import": {
+        "types": "./dist/query-engine/index.d.ts",
+        "default": "./dist/query-engine/index.js"
+      },
+      "default": {
+        "types": "./dist/query-engine/index.d.ts",
+        "default": "./dist/query-engine/index.js"
+      }
+    },
     "./llms": {
       "require": {
         "types": "./dist/llms/index.d.cts",
@@ -32,6 +60,20 @@
         "default": "./dist/decorator/index.js"
       }
     },
+    "./embeddings": {
+      "require": {
+        "types": "./dist/embeddings/index.d.cts",
+        "default": "./dist/embeddings/index.cjs"
+      },
+      "import": {
+        "types": "./dist/embeddings/index.d.ts",
+        "default": "./dist/embeddings/index.js"
+      },
+      "default": {
+        "types": "./dist/embeddings/index.d.ts",
+        "default": "./dist/embeddings/index.js"
+      }
+    },
     "./global": {
       "require": {
         "types": "./dist/global/index.d.cts",
@@ -89,7 +131,8 @@
   },
   "devDependencies": {
     "ajv": "^8.16.0",
-    "bunchee": "5.3.0-beta.0"
+    "bunchee": "5.3.1",
+    "natural": "^7.1.0"
   },
   "dependencies": {
     "@llamaindex/env": "workspace:*",

packages/core/src/embeddings/base.ts

@@ -1,9 +1,7 @@
-import type { MessageContentDetail } from "@llamaindex/core/llms";
-import type { BaseNode } from "@llamaindex/core/schema";
-import { MetadataMode } from "@llamaindex/core/schema";
-import { extractSingleText } from "@llamaindex/core/utils";
 import { type Tokenizers } from "@llamaindex/env";
-import type { TransformComponent } from "../ingestion/types.js";
+import type { MessageContentDetail } from "../llms";
+import { BaseNode, MetadataMode, TransformComponent } from "../schema";
+import { extractSingleText } from "../utils";
 import { truncateMaxTokens } from "./tokenizer.js";
 import { SimilarityType, similarity } from "./utils.js";
 
@@ -17,10 +15,33 @@ export type EmbeddingInfo = {
   tokenizer?: Tokenizers;
 };
 
-export abstract class BaseEmbedding implements TransformComponent {
+export type BaseEmbeddingOptions = {
+  logProgress?: boolean;
+};
+
+export abstract class BaseEmbedding extends TransformComponent {
   embedBatchSize = DEFAULT_EMBED_BATCH_SIZE;
   embedInfo?: EmbeddingInfo;
 
+  constructor() {
+    super(
+      async (
+        nodes: BaseNode[],
+        options?: BaseEmbeddingOptions,
+      ): Promise<BaseNode[]> => {
+        const texts = nodes.map((node) => node.getContent(MetadataMode.EMBED));
+
+        const embeddings = await this.getTextEmbeddingsBatch(texts, options);
+
+        for (let i = 0; i < nodes.length; i++) {
+          nodes[i].embedding = embeddings[i];
+        }
+
+        return nodes;
+      },
+    );
+  }
+
   similarity(
     embedding1: number[],
     embedding2: number[],
@@ -45,7 +66,7 @@ export abstract class BaseEmbedding implements TransformComponent {
    * Optionally override this method to retrieve multiple embeddings in a single request
    * @param texts
    */
-  async getTextEmbeddings(texts: string[]): Promise<Array<number[]>> {
+  getTextEmbeddings = async (texts: string[]): Promise<Array<number[]>> => {
     const embeddings: number[][] = [];
 
     for (const text of texts) {
@@ -54,7 +75,7 @@ export abstract class BaseEmbedding implements TransformComponent {
     }
 
     return embeddings;
-  }
+  };
 
   /**
    * Get embeddings for a batch of texts
@@ -63,30 +84,16 @@ export abstract class BaseEmbedding implements TransformComponent {
    */
   async getTextEmbeddingsBatch(
     texts: string[],
-    options?: {
-      logProgress?: boolean;
-    },
+    options?: BaseEmbeddingOptions,
   ): Promise<Array<number[]>> {
     return await batchEmbeddings(
       texts,
-      this.getTextEmbeddings.bind(this),
+      this.getTextEmbeddings,
       this.embedBatchSize,
       options,
     );
   }
 
-  async transform(nodes: BaseNode[], _options?: any): Promise<BaseNode[]> {
-    const texts = nodes.map((node) => node.getContent(MetadataMode.EMBED));
-
-    const embeddings = await this.getTextEmbeddingsBatch(texts, _options);
-
-    for (let i = 0; i < nodes.length; i++) {
-      nodes[i].embedding = embeddings[i];
-    }
-
-    return nodes;
-  }
-
   truncateMaxTokens(input: string[]): string[] {
     return input.map((s) => {
       // truncate to max tokens
@@ -104,9 +111,7 @@ export async function batchEmbeddings<T>(
   values: T[],
   embedFunc: EmbedFunc<T>,
   chunkSize: number,
-  options?: {
-    logProgress?: boolean;
-  },
+  options?: BaseEmbeddingOptions,
 ): Promise<Array<number[]>> {
   const resultEmbeddings: Array<number[]> = [];
 

packages/core/src/embeddings/index.ts

@@ -0,0 +1,4 @@
+export { BaseEmbedding, batchEmbeddings } from "./base";
+export type { BaseEmbeddingOptions, EmbeddingInfo } from "./base";
+export { truncateMaxTokens } from "./tokenizer";
+export { DEFAULT_SIMILARITY_TOP_K, SimilarityType, similarity } from "./utils";

packages/core/src/embeddings/utils.ts

@@ -0,0 +1,64 @@
+export const DEFAULT_SIMILARITY_TOP_K = 2;
+
+/**
+ * Similarity type
+ * Default is cosine similarity. Dot product and negative Euclidean distance are also supported.
+ */
+export enum SimilarityType {
+  DEFAULT = "cosine",
+  DOT_PRODUCT = "dot_product",
+  EUCLIDEAN = "euclidean",
+}
+
+/**
+ * The similarity between two embeddings.
+ * @param embedding1
+ * @param embedding2
+ * @param mode
+ * @returns similarity score with higher numbers meaning the two embeddings are more similar
+ */
+
+export function similarity(
+  embedding1: number[],
+  embedding2: number[],
+  mode: SimilarityType = SimilarityType.DEFAULT,
+): number {
+  if (embedding1.length !== embedding2.length) {
+    throw new Error("Embedding length mismatch");
+  }
+
+  // NOTE I've taken enough Kahan to know that we should probably leave the
+  // numeric programming to numeric programmers. The naive approach here
+  // will probably cause some avoidable loss of floating point precision
+  // ml-distance is worth watching although they currently also use the naive
+  // formulas
+  function norm(x: number[]): number {
+    let result = 0;
+    for (let i = 0; i < x.length; i++) {
+      result += x[i] * x[i];
+    }
+    return Math.sqrt(result);
+  }
+
+  switch (mode) {
+    case SimilarityType.EUCLIDEAN: {
+      const difference = embedding1.map((x, i) => x - embedding2[i]);
+      return -norm(difference);
+    }
+    case SimilarityType.DOT_PRODUCT: {
+      let result = 0;
+      for (let i = 0; i < embedding1.length; i++) {
+        result += embedding1[i] * embedding2[i];
+      }
+      return result;
+    }
+    case SimilarityType.DEFAULT: {
+      return (
+        similarity(embedding1, embedding2, SimilarityType.DOT_PRODUCT) /
+        (norm(embedding1) * norm(embedding2))
+      );
+    }
+    default:
+      throw new Error("Not implemented yet");
+  }
+}

packages/core/src/global/settings.ts

@@ -1,3 +1,4 @@
+import type { Tokenizer } from "@llamaindex/env";
 import {
   type CallbackManager,
   getCallbackManager,
@@ -9,8 +10,22 @@ import {
   setChunkSize,
   withChunkSize,
 } from "./settings/chunk-size";
+import {
+  getTokenizer,
+  setTokenizer,
+  withTokenizer,
+} from "./settings/tokenizer";
 
 export const Settings = {
+  get tokenizer() {
+    return getTokenizer();
+  },
+  set tokenizer(tokenizer) {
+    setTokenizer(tokenizer);
+  },
+  withTokenizer<Result>(tokenizer: Tokenizer, fn: () => Result): Result {
+    return withTokenizer(tokenizer, fn);
+  },
   get chunkSize(): number | undefined {
     return getChunkSize();
   },

packages/core/src/global/settings/callback-manager.ts

@@ -6,6 +6,7 @@ import type {
   ToolCall,
   ToolOutput,
 } from "../../llms";
+import { TextNode } from "../../schema";
 import { EventCaller, getEventCaller } from "../../utils/event-caller";
 import type { UUID } from "../type";
 
@@ -33,12 +34,32 @@ export type LLMStreamEvent = {
   chunk: ChatResponseChunk;
 };
 
+export type ChunkingStartEvent = {
+  text: string[];
+};
+
+export type ChunkingEndEvent = {
+  chunks: string[];
+};
+
+export type NodeParsingStartEvent = {
+  documents: TextNode[];
+};
+
+export type NodeParsingEndEvent = {
+  nodes: TextNode[];
+};
+
 export interface LlamaIndexEventMaps {
   "llm-start": LLMStartEvent;
   "llm-end": LLMEndEvent;
   "llm-tool-call": LLMToolCallEvent;
   "llm-tool-result": LLMToolResultEvent;
   "llm-stream": LLMStreamEvent;
+  "chunking-start": ChunkingStartEvent;
+  "chunking-end": ChunkingEndEvent;
+  "node-parsing-start": NodeParsingStartEvent;
+  "node-parsing-end": NodeParsingEndEvent;
 }
 
 export class LlamaIndexCustomEvent<T = any> extends CustomEvent<T> {
@@ -105,9 +126,7 @@ export class CallbackManager {
     }
     queueMicrotask(() => {
       cbs.forEach((handler) =>
-        handler(
-          LlamaIndexCustomEvent.fromEvent(event, structuredClone(detail)),
-        ),
+        handler(LlamaIndexCustomEvent.fromEvent(event, { ...detail })),
       );
     });
   }
@@ -118,14 +137,10 @@ export const globalCallbackManager = new CallbackManager();
 const callbackManagerAsyncLocalStorage =
   new AsyncLocalStorage<CallbackManager>();
 
-let currentCallbackManager: CallbackManager | null = null;
+let currentCallbackManager: CallbackManager = globalCallbackManager;
 
 export function getCallbackManager(): CallbackManager {
-  return (
-    callbackManagerAsyncLocalStorage.getStore() ??
-    currentCallbackManager ??
-    globalCallbackManager
-  );
+  return callbackManagerAsyncLocalStorage.getStore() ?? currentCallbackManager;
 }
 
 export function setCallbackManager(callbackManager: CallbackManager) {

packages/core/src/global/settings/chunk-size.ts

@@ -1,9 +1,9 @@
 import { AsyncLocalStorage } from "@llamaindex/env";
 
 const chunkSizeAsyncLocalStorage = new AsyncLocalStorage<number | undefined>();
-let globalChunkSize: number | null = null;
+let globalChunkSize: number = 1024;
 
-export function getChunkSize(): number | undefined {
+export function getChunkSize(): number {
   return globalChunkSize ?? chunkSizeAsyncLocalStorage.getStore();
 }
 

packages/core/src/global/settings/tokenizer.ts

@@ -0,0 +1,21 @@
+import { AsyncLocalStorage, type Tokenizer, tokenizers } from "@llamaindex/env";
+
+const chunkSizeAsyncLocalStorage = new AsyncLocalStorage<Tokenizer>();
+let globalTokenizer: Tokenizer = tokenizers.tokenizer();
+
+export function getTokenizer(): Tokenizer {
+  return globalTokenizer ?? chunkSizeAsyncLocalStorage.getStore();
+}
+
+export function setTokenizer(tokenizer: Tokenizer | undefined) {
+  if (tokenizer !== undefined) {
+    globalTokenizer = tokenizer;
+  }
+}
+
+export function withTokenizer<Result>(
+  tokenizer: Tokenizer,
+  fn: () => Result,
+): Result {
+  return chunkSizeAsyncLocalStorage.run(tokenizer, fn);
+}

packages/core/src/node-parser/base.ts

@@ -0,0 +1,149 @@
+import { Settings } from "../global";
+import {
+  BaseNode,
+  buildNodeFromSplits,
+  MetadataMode,
+  NodeRelationship,
+  TextNode,
+  TransformComponent,
+} from "../schema";
+
+export abstract class NodeParser extends TransformComponent {
+  includeMetadata: boolean = true;
+  includePrevNextRel: boolean = true;
+
+  constructor() {
+    super(async (nodes: BaseNode[]): Promise<BaseNode[]> => {
+      return this.getNodesFromDocuments(nodes as TextNode[]);
+    });
+  }
+
+  protected postProcessParsedNodes(
+    nodes: TextNode[],
+    parentDocMap: Map<string, TextNode>,
+  ): TextNode[] {
+    nodes.forEach((node, i) => {
+      const parentDoc = parentDocMap.get(node.sourceNode?.nodeId || "");
+
+      if (parentDoc) {
+        const startCharIdx = parentDoc.text.indexOf(
+          node.getContent(MetadataMode.NONE),
+        );
+        if (startCharIdx >= 0) {
+          node.startCharIdx = startCharIdx;
+          node.endCharIdx =
+            startCharIdx + node.getContent(MetadataMode.NONE).length;
+        }
+        if (this.includeMetadata && node.metadata && parentDoc.metadata) {
+          node.metadata = { ...node.metadata, ...parentDoc.metadata };
+        }
+      }
+
+      if (this.includePrevNextRel && node.sourceNode) {
+        const previousNode = i > 0 ? nodes[i - 1] : null;
+        const nextNode = i < nodes.length - 1 ? nodes[i + 1] : null;
+
+        if (
+          previousNode &&
+          previousNode.sourceNode &&
+          previousNode.sourceNode.nodeId === node.sourceNode.nodeId
+        ) {
+          node.relationships = {
+            ...node.relationships,
+            [NodeRelationship.PREVIOUS]: previousNode.asRelatedNodeInfo(),
+          };
+        }
+
+        if (
+          nextNode &&
+          nextNode.sourceNode &&
+          nextNode.sourceNode.nodeId === node.sourceNode.nodeId
+        ) {
+          node.relationships = {
+            ...node.relationships,
+            [NodeRelationship.NEXT]: nextNode.asRelatedNodeInfo(),
+          };
+        }
+      }
+    });
+
+    return nodes;
+  }
+
+  protected abstract parseNodes(
+    documents: TextNode[],
+    showProgress?: boolean,
+  ): TextNode[];
+
+  public getNodesFromDocuments(documents: TextNode[]): TextNode[] {
+    const docsId: Map<string, TextNode> = new Map(
+      documents.map((doc) => [doc.id_, doc]),
+    );
+    const callbackManager = Settings.callbackManager;
+
+    callbackManager.dispatchEvent("node-parsing-start", {
+      documents,
+    });
+
+    const nodes = this.postProcessParsedNodes(
+      this.parseNodes(documents),
+      docsId,
+    );
+
+    callbackManager.dispatchEvent("node-parsing-end", {
+      nodes,
+    });
+
+    return nodes;
+  }
+}
+
+export abstract class TextSplitter extends NodeParser {
+  abstract splitText(text: string): string[];
+
+  public splitTexts(texts: string[]): string[] {
+    return texts.flatMap((text) => this.splitText(text));
+  }
+
+  protected parseNodes(nodes: TextNode[]): TextNode[] {
+    return nodes.reduce<TextNode[]>((allNodes, node) => {
+      const splits = this.splitText(node.getContent(MetadataMode.ALL));
+      const nodes = buildNodeFromSplits(splits, node);
+      return allNodes.concat(nodes);
+    }, []);
+  }
+}
+
+export abstract class MetadataAwareTextSplitter extends TextSplitter {
+  abstract splitTextMetadataAware(text: string, metadata: string): string[];
+
+  splitTextsMetadataAware(texts: string[], metadata: string[]): string[] {
+    if (texts.length !== metadata.length) {
+      throw new TypeError("`texts` and `metadata` must have the same length");
+    }
+    return texts.flatMap((text, i) =>
+      this.splitTextMetadataAware(text, metadata[i]),
+    );
+  }
+
+  protected getMetadataString(node: TextNode): string {
+    const embedStr = node.getMetadataStr(MetadataMode.EMBED);
+    const llmStr = node.getMetadataStr(MetadataMode.LLM);
+    if (embedStr.length > llmStr.length) {
+      return embedStr;
+    } else {
+      return llmStr;
+    }
+  }
+
+  protected parseNodes(nodes: TextNode[]): TextNode[] {
+    return nodes.reduce<TextNode[]>((allNodes, node) => {
+      const metadataStr = this.getMetadataString(node);
+      const splits = this.splitTextMetadataAware(
+        node.getContent(MetadataMode.NONE),
+        metadataStr,
+      );
+      return allNodes.concat(buildNodeFromSplits(splits, node));
+    }, []);
+  }
+}

packages/core/src/node-parser/index.ts

@@ -0,0 +1,24 @@
+/**
+ * Current logic is based on the following implementation:
+ * @link @link https://github.com/run-llama/llama_index/blob/cc0ea90e7e72b8e4f5069aac981d56bb1d568323/llama-index-core/llama_index/core/node_parser
+ */
+import { SentenceSplitter } from "./sentence-splitter";
+
+/**
+ * @deprecated Use `SentenceSplitter` instead
+ */
+export const SimpleNodeParser = SentenceSplitter;
+
+export { MetadataAwareTextSplitter, NodeParser, TextSplitter } from "./base";
+export { MarkdownNodeParser } from "./markdown";
+export { SentenceSplitter } from "./sentence-splitter";
+export { SentenceWindowNodeParser } from "./sentence-window";
+export type { SplitterParams } from "./type";
+export {
+  splitByChar,
+  splitByPhraseRegex,
+  splitByRegex,
+  splitBySentenceTokenizer,
+  splitBySep,
+} from "./utils";
+export type { TextSplitterFn } from "./utils";

packages/core/src/node-parser/markdown.ts

@@ -0,0 +1,87 @@
+import {
+  buildNodeFromSplits,
+  type Metadata,
+  MetadataMode,
+  TextNode,
+} from "../schema";
+import { NodeParser } from "./base";
+
+export class MarkdownNodeParser extends NodeParser {
+  override parseNodes(nodes: TextNode[], showProgress?: boolean): TextNode[] {
+    return nodes.reduce<TextNode[]>((allNodes, node) => {
+      const markdownNodes = this.getNodesFromNode(node);
+      return allNodes.concat(markdownNodes);
+    }, []);
+  }
+
+  protected getNodesFromNode(node: TextNode): TextNode[] {
+    const text = node.getContent(MetadataMode.NONE);
+    const markdownNodes: TextNode[] = [];
+    const lines = text.split("\n");
+    let metadata: { [key: string]: string } = {};
+    let codeBlock = false;
+    let currentSection = "";
+
+    for (const line of lines) {
+      if (line.trim().startsWith("```")) {
+        codeBlock = !codeBlock;
+      }
+      const headerMatch = /^(#+)\s(.*)/.exec(line);
+      if (headerMatch && !codeBlock) {
+        if (currentSection !== "") {
+          markdownNodes.push(
+            this.buildNodeFromSplit(currentSection.trim(), node, metadata),
+          );
+        }
+        metadata = this.updateMetadata(
+          metadata,
+          headerMatch[2],
+          headerMatch[1].trim().length,
+        );
+        currentSection = `${headerMatch[2]}\n`;
+      } else {
+        currentSection += line + "\n";
+      }
+    }
+
+    if (currentSection !== "") {
+      markdownNodes.push(
+        this.buildNodeFromSplit(currentSection.trim(), node, metadata),
+      );
+    }
+
+    return markdownNodes;
+  }
+
+  private updateMetadata(
+    headersMetadata: { [key: string]: string },
+    newHeader: string,
+    newHeaderLevel: number,
+  ): { [key: string]: string } {
+    const updatedHeaders: { [key: string]: string } = {};
+
+    for (let i = 1; i < newHeaderLevel; i++) {
+      const key = `Header_${i}`;
+      if (key in headersMetadata) {
+        updatedHeaders[key] = headersMetadata[key];
+      }
+    }
+
+    updatedHeaders[`Header_${newHeaderLevel}`] = newHeader;
+    return updatedHeaders;
+  }
+
+  private buildNodeFromSplit(
+    textSplit: string,
+    node: TextNode,
+    metadata: Metadata,
+  ): TextNode {
+    const newNode = buildNodeFromSplits([textSplit], node, undefined)[0];
+
+    if (this.includeMetadata) {
+      newNode.metadata = { ...newNode.metadata, ...metadata };
+    }
+
+    return newNode;
+  }
+}

packages/core/src/node-parser/sentence-splitter.ts

@@ -0,0 +1,226 @@
+import type { Tokenizer } from "@llamaindex/env";
+import { z } from "zod";
+import { Settings } from "../global";
+import { sentenceSplitterSchema } from "../schema";
+import { MetadataAwareTextSplitter } from "./base";
+import type { SplitterParams } from "./type";
+import {
+  splitByChar,
+  splitByRegex,
+  splitBySentenceTokenizer,
+  splitBySep,
+  type TextSplitterFn,
+} from "./utils";
+
+type _Split = {
+  text: string;
+  isSentence: boolean;
+  tokenSize: number;
+};
+
+/**
+ * Parse text with a preference for complete sentences.
+ */
+export class SentenceSplitter extends MetadataAwareTextSplitter {
+  /**
+   * The token chunk size for each chunk.
+   */
+  chunkSize: number = 1024;
+  /**
+   * The token overlap of each chunk when splitting.
+   */
+  chunkOverlap: number = 200;
+  /**
+   * Default separator for splitting into words
+   */
+  separator: string = " ";
+  /**
+   * Separator between paragraphs.
+   */
+  paragraphSeparator: string = "\n\n\n";
+  /**
+   * Backup regex for splitting into sentences.
+   */
+  secondaryChunkingRegex: string = "[^,.;。？！]+[,.;。？！]?";
+
+  #chunkingTokenizerFn = splitBySentenceTokenizer();
+  #splitFns: Set<TextSplitterFn> = new Set();
+  #subSentenceSplitFns: Set<TextSplitterFn> = new Set();
+  #tokenizer: Tokenizer;
+
+  constructor(
+    params?: z.input<typeof sentenceSplitterSchema> & SplitterParams,
+  ) {
+    super();
+    if (params) {
+      const parsedParams = sentenceSplitterSchema.parse(params);
+      this.chunkSize = parsedParams.chunkSize;
+      this.chunkOverlap = parsedParams.chunkOverlap;
+      this.separator = parsedParams.separator;
+      this.paragraphSeparator = parsedParams.paragraphSeparator;
+      this.secondaryChunkingRegex = parsedParams.secondaryChunkingRegex;
+    }
+    this.#tokenizer = params?.tokenizer ?? Settings.tokenizer;
+    this.#splitFns.add(splitBySep(this.paragraphSeparator));
+    this.#splitFns.add(this.#chunkingTokenizerFn);
+
+    this.#subSentenceSplitFns.add(splitByRegex(this.secondaryChunkingRegex));
+    this.#subSentenceSplitFns.add(splitBySep(this.separator));
+    this.#subSentenceSplitFns.add(splitByChar());
+  }
+
+  splitTextMetadataAware(text: string, metadata: string): string[] {
+    const metadataLength = this.tokenSize(metadata);
+    const effectiveChunkSize = this.chunkSize - metadataLength;
+    if (effectiveChunkSize <= 0) {
+      throw new Error(
+        `Metadata length (${metadataLength}) is longer than chunk size (${this.chunkSize}). Consider increasing the chunk size or decreasing the size of your metadata to avoid this.`,
+      );
+    } else if (effectiveChunkSize < 50) {
+      console.log(
+        `Metadata length (${metadataLength}) is close to chunk size (${this.chunkSize}). Resulting chunks are less than 50 tokens. Consider increasing the chunk size or decreasing the size of your metadata to avoid this.`,
+      );
+    }
+    return this._splitText(text, effectiveChunkSize);
+  }
+
+  splitText(text: string): string[] {
+    return this._splitText(text, this.chunkSize);
+  }
+
+  _splitText(text: string, chunkSize: number): string[] {
+    if (text === "") return [text];
+
+    const callbackManager = Settings.callbackManager;
+
+    callbackManager.dispatchEvent("chunking-start", {
+      text: [text],
+    });
+    const splits = this.#split(text, chunkSize);
+    const chunks = this.#merge(splits, chunkSize);
+
+    callbackManager.dispatchEvent("chunking-end", {
+      chunks,
+    });
+    return chunks;
+  }
+
+  #split(text: string, chunkSize: number): _Split[] {
+    const tokenSize = this.tokenSize(text);
+    if (tokenSize <= chunkSize) {
+      return [
+        {
+          text,
+          isSentence: true,
+          tokenSize,
+        },
+      ];
+    }
+    const [textSplitsByFns, isSentence] = this.#getSplitsByFns(text);
+    const textSplits: _Split[] = [];
+
+    for (const textSplit of textSplitsByFns) {
+      const tokenSize = this.tokenSize(textSplit);
+      if (tokenSize <= chunkSize) {
+        textSplits.push({
+          text: textSplit,
+          isSentence,
+          tokenSize,
+        });
+      } else {
+        const recursiveTextSplits = this.#split(textSplit, chunkSize);
+        textSplits.push(...recursiveTextSplits);
+      }
+    }
+    return textSplits;
+  }
+
+  #getSplitsByFns(text: string): [splits: string[], isSentence: boolean] {
+    for (const splitFn of this.#splitFns) {
+      const splits = splitFn(text);
+      if (splits.length > 1) {
+        return [splits, true];
+      }
+    }
+    for (const splitFn of this.#subSentenceSplitFns) {
+      const splits = splitFn(text);
+      if (splits.length > 1) {
+        return [splits, false];
+      }
+    }
+    return [[text], true];
+  }
+
+  #merge(splits: _Split[], chunkSize: number): string[] {
+    const chunks: string[] = [];
+    let currentChunk: [string, number][] = [];
+    let lastChunk: [string, number][] = [];
+    let currentChunkLength = 0;
+    let newChunk = true;
+
+    const closeChunk = (): void => {
+      chunks.push(currentChunk.map(([text]) => text).join(""));
+      lastChunk = currentChunk;
+      currentChunk = [];
+      currentChunkLength = 0;
+      newChunk = true;
+
+      let lastIndex = lastChunk.length - 1;
+      while (
+        lastIndex >= 0 &&
+        currentChunkLength + lastChunk[lastIndex][1] <= this.chunkOverlap
+      ) {
+        const [text, length] = lastChunk[lastIndex];
+        currentChunkLength += length;
+        currentChunk.unshift([text, length]);
+        lastIndex -= 1;
+      }
+    };
+
+    while (splits.length > 0) {
+      const curSplit = splits[0];
+      if (curSplit.tokenSize > chunkSize) {
+        throw new Error("Single token exceeded chunk size");
+      }
+      if (currentChunkLength + curSplit.tokenSize > chunkSize && !newChunk) {
+        closeChunk();
+      } else {
+        if (
+          curSplit.isSentence ||
+          currentChunkLength + curSplit.tokenSize <= chunkSize ||
+          newChunk
+        ) {
+          currentChunkLength += curSplit.tokenSize;
+          currentChunk.push([curSplit.text, curSplit.tokenSize]);
+          splits.shift();
+          newChunk = false;
+        } else {
+          closeChunk();
+        }
+      }
+    }
+
+    // Handle the last chunk
+    if (!newChunk) {
+      chunks.push(currentChunk.map(([text]) => text).join(""));
+    }
+
+    return this.#postprocessChunks(chunks);
+  }
+
+  /**
+   * Remove whitespace only chunks and remove leading and trailing whitespace.
+   */
+  #postprocessChunks(chunks: string[]): string[] {
+    const newChunks: string[] = [];
+    for (const chunk of chunks) {
+      const trimmedChunk = chunk.trim();
+      if (trimmedChunk !== "") {
+        newChunks.push(trimmedChunk);
+      }
+    }
+    return newChunks;
+  }
+
+  tokenSize = (text: string) => this.#tokenizer.encode(text).length;
+}

packages/core/src/node-parser/sentence-tokenizer-parser.d.ts

@@ -0,0 +1,5 @@
+declare class SentenceTokenizer {
+  tokenize(text: string): string[];
+}
+
+export { SentenceTokenizer as default };

packages/core/src/node-parser/sentence-window.ts

@@ -0,0 +1,85 @@
+import { randomUUID } from "@llamaindex/env";
+import { z } from "zod";
+import {
+  buildNodeFromSplits,
+  Document,
+  sentenceWindowNodeParserSchema,
+  TextNode,
+} from "../schema";
+import { NodeParser } from "./base";
+import { splitBySentenceTokenizer, type TextSplitterFn } from "./utils";
+
+export class SentenceWindowNodeParser extends NodeParser {
+  static DEFAULT_WINDOW_SIZE = 3;
+  static DEFAULT_WINDOW_METADATA_KEY = "window";
+  static DEFAULT_ORIGINAL_TEXT_METADATA_KEY = "originalText";
+
+  windowSize: number;
+  windowMetadataKey: string;
+  originalTextMetadataKey: string;
+  sentenceSplitter: TextSplitterFn = splitBySentenceTokenizer();
+  idGenerator: () => string = () => randomUUID();
+
+  constructor(params?: z.input<typeof sentenceWindowNodeParserSchema>) {
+    super();
+    if (params) {
+      const parsedParams = sentenceWindowNodeParserSchema.parse(params);
+      this.windowSize = parsedParams.windowSize;
+      this.windowMetadataKey = parsedParams.windowMetadataKey;
+      this.originalTextMetadataKey = parsedParams.originalTextMetadataKey;
+    } else {
+      this.windowSize = SentenceWindowNodeParser.DEFAULT_WINDOW_SIZE;
+      this.windowMetadataKey =
+        SentenceWindowNodeParser.DEFAULT_WINDOW_METADATA_KEY;
+      this.originalTextMetadataKey =
+        SentenceWindowNodeParser.DEFAULT_ORIGINAL_TEXT_METADATA_KEY;
+    }
+  }
+
+  override parseNodes(nodes: TextNode[], showProgress?: boolean): TextNode[] {
+    return nodes.reduce<TextNode[]>((allNodes, node) => {
+      const nodes = this.buildWindowNodesFromDocuments([node]);
+      return allNodes.concat(nodes);
+    }, []);
+  }
+
+  buildWindowNodesFromDocuments(documents: Document[]): TextNode[] {
+    const allNodes: TextNode[] = [];
+
+    for (const doc of documents) {
+      const text = doc.text;
+      const textSplits = this.sentenceSplitter(text);
+      const nodes = buildNodeFromSplits(
+        textSplits,
+        doc,
+        undefined,
+        this.idGenerator,
+      );
+
+      nodes.forEach((node, i) => {
+        const windowNodes = nodes.slice(
+          Math.max(0, i - this.windowSize),
+          Math.min(i + this.windowSize + 1, nodes.length),
+        );
+
+        node.metadata[this.windowMetadataKey] = windowNodes
+          .map((n) => n.text)
+          .join(" ");
+        node.metadata[this.originalTextMetadataKey] = node.text;
+
+        node.excludedEmbedMetadataKeys.push(
+          this.windowMetadataKey,
+          this.originalTextMetadataKey,
+        );
+        node.excludedLlmMetadataKeys.push(
+          this.windowMetadataKey,
+          this.originalTextMetadataKey,
+        );
+      });
+
+      allNodes.push(...nodes);
+    }
+
+    return allNodes;
+  }
+}

packages/core/src/node-parser/type.ts

@@ -0,0 +1,5 @@
+import type { Tokenizer } from "@llamaindex/env";
+
+export type SplitterParams = {
+  tokenizer?: Tokenizer;
+};

packages/core/src/node-parser/utils.ts

@@ -0,0 +1,57 @@
+import type { TextSplitter } from "./base";
+import SentenceTokenizerNew from "./sentence-tokenizer-parser.js";
+
+export type TextSplitterFn = (text: string) => string[];
+
+const truncateText = (text: string, textSplitter: TextSplitter): string => {
+  const chunks = textSplitter.splitText(text);
+  return chunks[0];
+};
+
+const splitTextKeepSeparator = (text: string, separator: string): string[] => {
+  const parts = text.split(separator);
+  const result = parts.map((part, index) =>
+    index > 0 ? separator + part : part,
+  );
+  return result.filter((s) => s);
+};
+
+export const splitBySep = (
+  sep: string,
+  keepSep: boolean = true,
+): TextSplitterFn => {
+  if (keepSep) {
+    return (text: string) => splitTextKeepSeparator(text, sep);
+  } else {
+    return (text: string) => text.split(sep);
+  }
+};
+
+export const splitByChar = (): TextSplitterFn => {
+  return (text: string) => text.split("");
+};
+
+let sentenceTokenizer: SentenceTokenizerNew | null = null;
+
+export const splitBySentenceTokenizer = (): TextSplitterFn => {
+  if (!sentenceTokenizer) {
+    sentenceTokenizer = new SentenceTokenizerNew();
+  }
+  const tokenizer = sentenceTokenizer;
+  return (text: string) => {
+    try {
+      return tokenizer.tokenize(text);
+    } catch {
+      return [text];
+    }
+  };
+};
+
+export const splitByRegex = (regex: string): TextSplitterFn => {
+  return (text: string) => text.match(new RegExp(regex, "g")) || [];
+};
+
+export const splitByPhraseRegex = (): TextSplitterFn => {
+  const regex = "[^,.;]+[,.;]?";
+  return splitByRegex(regex);
+};

packages/core/src/query-engine/base.ts

@@ -0,0 +1,29 @@
+import type { MessageContent } from "../llms";
+import { EngineResponse, type NodeWithScore } from "../schema";
+
+/**
+ * @link https://docs.llamaindex.ai/en/stable/api_reference/schema/?h=querybundle#llama_index.core.schema.QueryBundle
+ *
+ *  We don't have `image_path` here, because it is included in the `query` field.
+ */
+export type QueryBundle = {
+  query: MessageContent;
+  customEmbeddings?: string[];
+  embeddings?: number[];
+};
+
+export type QueryType = string | QueryBundle;
+
+export interface BaseQueryEngine {
+  query(
+    strOrQueryBundle: QueryType,
+    stream: true,
+  ): Promise<AsyncIterable<EngineResponse>>;
+  query(strOrQueryBundle: QueryType, stream?: false): Promise<EngineResponse>;
+
+  synthesize?(
+    strOrQueryBundle: QueryType,
+    nodes: NodeWithScore[],
+    additionalSources?: Iterator<NodeWithScore>,
+  ): Promise<EngineResponse>;
+}

packages/core/src/query-engine/index.ts

@@ -0,0 +1 @@
+export type { BaseQueryEngine, QueryBundle, QueryType } from "./base";

packages/core/src/schema/index.ts

@@ -1,2 +1,4 @@
 export * from "./node";
+export { FileReader, TransformComponent, type BaseReader } from "./type";
+export { EngineResponse } from "./type/engine–response";
 export * from "./zod";

packages/core/src/schema/node.ts

@@ -450,3 +450,48 @@ export function splitNodesByType(nodes: BaseNode[]): NodesByType {
   }
   return result;
 }
+
+export function buildNodeFromSplits(
+  textSplits: string[],
+  doc: BaseNode,
+  refDoc: BaseNode = doc,
+  idGenerator: (idx: number, refDoc: BaseNode) => string = () => randomUUID(),
+) {
+  const nodes: TextNode[] = [];
+  const relationships = {
+    [NodeRelationship.SOURCE]: refDoc.asRelatedNodeInfo(),
+  };
+
+  textSplits.forEach((textChunk, i) => {
+    if (doc instanceof ImageDocument) {
+      const imageNode = new ImageNode({
+        id_: idGenerator(i, doc),
+        text: textChunk,
+        image: doc.image,
+        embedding: doc.embedding,
+        excludedEmbedMetadataKeys: [...doc.excludedEmbedMetadataKeys],
+        excludedLlmMetadataKeys: [...doc.excludedLlmMetadataKeys],
+        metadataSeparator: doc.metadataSeparator,
+        textTemplate: doc.textTemplate,
+        relationships: { ...relationships },
+      });
+      nodes.push(imageNode);
+    } else if (doc instanceof Document || doc instanceof TextNode) {
+      const node = new TextNode({
+        id_: idGenerator(i, doc),
+        text: textChunk,
+        embedding: doc.embedding,
+        excludedEmbedMetadataKeys: [...doc.excludedEmbedMetadataKeys],
+        excludedLlmMetadataKeys: [...doc.excludedLlmMetadataKeys],
+        metadataSeparator: doc.metadataSeparator,
+        textTemplate: doc.textTemplate,
+        relationships: { ...relationships },
+      });
+      nodes.push(node);
+    } else {
+      throw new Error(`Unknown document type: ${doc.type}`);
+    }
+  });
+
+  return nodes;
+}

packages/core/src/schema/type.ts

@@ -0,0 +1,64 @@
+import { fs, path, randomUUID } from "@llamaindex/env";
+import type { BaseNode, Document } from "./node";
+
+interface TransformComponentSignature {
+  <Options extends Record<string, unknown>>(
+    nodes: BaseNode[],
+    options?: Options,
+  ): Promise<BaseNode[]>;
+}
+
+export interface TransformComponent extends TransformComponentSignature {
+  id: string;
+}
+
+export class TransformComponent {
+  constructor(transformFn: TransformComponentSignature) {
+    Object.defineProperties(
+      transformFn,
+      Object.getOwnPropertyDescriptors(this.constructor.prototype),
+    );
+    const transform = function transform(
+      ...args: Parameters<TransformComponentSignature>
+    ) {
+      return transformFn(...args);
+    };
+    Reflect.setPrototypeOf(transform, new.target.prototype);
+    transform.id = randomUUID();
+    return transform;
+  }
+}
+
+/**
+ * A reader takes imports data into Document objects.
+ */
+export interface BaseReader {
+  loadData(...args: unknown[]): Promise<Document[]>;
+}
+
+/**
+ * A FileReader takes file paths and imports data into Document objects.
+ */
+export abstract class FileReader implements BaseReader {
+  abstract loadDataAsContent(
+    fileContent: Uint8Array,
+    fileName?: string,
+  ): Promise<Document[]>;
+
+  async loadData(filePath: string): Promise<Document[]> {
+    const fileContent = await fs.readFile(filePath);
+    const fileName = path.basename(filePath);
+    const docs = await this.loadDataAsContent(fileContent, fileName);
+    docs.forEach(FileReader.addMetaData(filePath));
+    return docs;
+  }
+
+  static addMetaData(filePath: string) {
+    return (doc: Document, index: number) => {
+      // generate id as loadDataAsContent is only responsible for the content
+      doc.id_ = `${filePath}_${index + 1}`;
+      doc.metadata["file_path"] = path.resolve(filePath);
+      doc.metadata["file_name"] = path.basename(filePath);
+    };
+  }
+}

packages/core/src/schema/type/engine–response.ts

@@ -1,20 +1,16 @@
-import type {
-  ChatMessage,
-  ChatResponse,
-  ChatResponseChunk,
-} from "@llamaindex/core/llms";
-import type { NodeWithScore } from "@llamaindex/core/schema";
-import { extractText } from "@llamaindex/core/utils";
+import type { ChatMessage, ChatResponse, ChatResponseChunk } from "../../llms";
+import { extractText } from "../../utils";
+import type { Metadata, NodeWithScore } from "../node";
 
 export class EngineResponse implements ChatResponse, ChatResponseChunk {
   sourceNodes?: NodeWithScore[];
 
-  metadata: Record<string, unknown> = {};
+  metadata: Metadata = {};
 
   message: ChatMessage;
   raw: object | null;
 
-  #stream: boolean;
+  readonly stream: boolean;
 
   private constructor(
     chatResponse: ChatResponse,
@@ -24,7 +20,7 @@ export class EngineResponse implements ChatResponse, ChatResponseChunk {
     this.message = chatResponse.message;
     this.raw = chatResponse.raw;
     this.sourceNodes = sourceNodes;
-    this.#stream = stream;
+    this.stream = stream;
   }
 
   static fromResponse(
@@ -70,13 +66,15 @@ export class EngineResponse implements ChatResponse, ChatResponseChunk {
     );
   }
 
-  // @deprecated use 'message' instead
+  /**
+   * @deprecated Use `message` instead.
+   */
   get response(): string {
     return extractText(this.message.content);
   }
 
   get delta(): string {
-    if (!this.#stream) {
+    if (!this.stream) {
       console.warn(
         "delta is only available for streaming responses. Consider using 'message' instead.",
       );
@@ -84,7 +82,7 @@ export class EngineResponse implements ChatResponse, ChatResponseChunk {
     return extractText(this.message.content);
   }
 
-  toString() {
+  toString(): string {
     return this.response ?? "";
   }
 }

packages/core/src/schema/zod.ts

@@ -1,4 +1,5 @@
 import { z } from "zod";
+import { Settings } from "../global";
 
 export const anyFunctionSchema = z.function(z.tuple([]).rest(z.any()), z.any());
 
@@ -16,3 +17,62 @@ export const baseToolSchema = z.object({
 export const baseToolWithCallSchema = baseToolSchema.extend({
   call: z.function(),
 });
+
+export const sentenceSplitterSchema = z
+  .object({
+    chunkSize: z
+      .number({
+        description: "The token chunk size for each chunk.",
+      })
+      .gt(0)
+      .optional()
+      .default(() => Settings.chunkSize ?? 1024),
+    chunkOverlap: z
+      .number({
+        description: "The token overlap of each chunk when splitting.",
+      })
+      .gte(0)
+      .optional()
+      .default(200),
+    separator: z
+      .string({
+        description: "Default separator for splitting into words",
+      })
+      .default(" "),
+    paragraphSeparator: z
+      .string({
+        description: "Separator between paragraphs.",
+      })
+      .optional()
+      .default("\n\n\n"),
+    secondaryChunkingRegex: z
+      .string({
+        description: "Backup regex for splitting into sentences.",
+      })
+      .optional()
+      .default("[^,.;。？！]+[,.;。？！]?"),
+  })
+  .refine(
+    (data) => data.chunkOverlap < data.chunkSize,
+    "Chunk overlap must be less than chunk size.",
+  );
+
+export const sentenceWindowNodeParserSchema = z.object({
+  windowSize: z
+    .number({
+      description:
+        "The number of sentences on each side of a sentence to capture.",
+    })
+    .gt(0)
+    .default(3),
+  windowMetadataKey: z
+    .string({
+      description: "The metadata key to store the sentence window under.",
+    })
+    .default("window"),
+  originalTextMetadataKey: z
+    .string({
+      description: "The metadata key to store the original sentence in.",
+    })
+    .default("originalText"),
+});

packages/core/src/utils/llms.ts

@@ -3,15 +3,22 @@ import type {
   MessageContentDetail,
   MessageContentTextDetail,
 } from "../llms";
+import type { QueryType } from "../query-engine";
 import type { ImageType } from "../schema";
 
 /**
- * Extracts just the text from a multi-modal message or the message itself if it's just text.
+ * Extracts just the text whether from
+ *  a multi-modal message
+ *  a single text message
+ *  or a query
  *
  * @param message The message to extract text from.
  * @returns The extracted text
  */
-export function extractText(message: MessageContent): string {
+export function extractText(message: MessageContent | QueryType): string {
+  if (typeof message === "object" && "query" in message) {
+    return extractText(message.query);
+  }
   if (typeof message !== "string" && !Array.isArray(message)) {
     console.warn(
       "extractText called with non-MessageContent message, this is likely a bug.",

packages/core/src/utils/wrap-llm-event.ts

@@ -1,5 +1,5 @@
 import { AsyncLocalStorage, randomUUID } from "@llamaindex/env";
-import { getCallbackManager } from "../global/settings/callback-manager";
+import { Settings } from "../global";
 import type { ChatResponse, ChatResponseChunk, LLM, LLMChat } from "../llms";
 
 export function wrapLLMEvent<
@@ -21,7 +21,7 @@ export function wrapLLMEvent<
     LLMChat<AdditionalChatOptions, AdditionalMessageOptions>["chat"]
   > {
     const id = randomUUID();
-    getCallbackManager().dispatchEvent("llm-start", {
+    Settings.callbackManager.dispatchEvent("llm-start", {
       id,
       messages: params[0].messages,
     });
@@ -55,7 +55,7 @@ export function wrapLLMEvent<
               ...chunk.options,
             };
           }
-          getCallbackManager().dispatchEvent("llm-stream", {
+          Settings.callbackManager.dispatchEvent("llm-stream", {
             id,
             chunk,
           });
@@ -63,14 +63,14 @@ export function wrapLLMEvent<
           yield chunk;
         }
         snapshot(() => {
-          getCallbackManager().dispatchEvent("llm-end", {
+          Settings.callbackManager.dispatchEvent("llm-end", {
             id,
             response: finalResponse,
           });
         });
       };
     } else {
-      getCallbackManager().dispatchEvent("llm-end", {
+      Settings.callbackManager.dispatchEvent("llm-end", {
         id,
         response,
       });

packages/core/tests/embeddings.test.ts

@@ -1,6 +1,6 @@
+import { truncateMaxTokens } from "@llamaindex/core/embeddings";
 import { Tokenizers, tokenizers } from "@llamaindex/env";
 import { describe, expect, test } from "vitest";
-import { truncateMaxTokens } from "../../src/embeddings/tokenizer.js";
 
 describe("truncateMaxTokens", () => {
   const tokenizer = tokenizers.tokenizer(Tokenizers.CL100K_BASE);

packages/core/tests/event-system.test.ts

@@ -6,6 +6,9 @@ declare module "@llamaindex/core/global" {
     test: {
       value: number;
     };
+    functionTest: {
+      fn: ({ x }: { x: number }) => string;
+    };
   }
 }
 
@@ -42,6 +45,25 @@ describe("event system", () => {
     expect(callback).toHaveBeenCalledTimes(1);
   });
 
+  test("dispatch function tool event", async () => {
+    const testFunction = ({ x }: { x: number }) => `${x * 2}`;
+    let callback;
+    Settings.callbackManager.on(
+      "functionTest",
+      (callback = vi.fn((event) => {
+        const data = event.detail;
+        expect(data.fn).toBe(testFunction);
+      })),
+    );
+
+    Settings.callbackManager.dispatchEvent("functionTest", {
+      fn: testFunction,
+    });
+    expect(callback).toHaveBeenCalledTimes(0);
+    await new Promise((resolve) => process.nextTick(resolve));
+    expect(callback).toHaveBeenCalledTimes(1);
+  });
+
   // rollup doesn't support decorators for now
   // test('wrap event caller', async () => {
   //   class A {

packages/core/tests/node-parser/markdown.test.ts

@@ -1,5 +1,5 @@
+import { MarkdownNodeParser } from "@llamaindex/core/node-parser";
 import { Document, MetadataMode } from "@llamaindex/core/schema";
-import { MarkdownNodeParser } from "llamaindex/nodeParsers/index";
 import { describe, expect, test } from "vitest";
 
 describe("MarkdownNodeParser", () => {
@@ -19,8 +19,8 @@ Header 2 content
     ]);
 
     expect(splits.length).toBe(2);
-    expect(splits[0].metadata).toEqual({ "Header 1": "Main Header" });
-    expect(splits[1].metadata).toEqual({ "Header 1": "Header 2" });
+    expect(splits[0].metadata).toEqual({ Header_1: "Main Header" });
+    expect(splits[1].metadata).toEqual({ Header_1: "Header 2" });
     expect(splits[0].getContent(MetadataMode.NONE)).toStrictEqual(
       "Main Header\n\nHeader 1 content",
     );
@@ -89,16 +89,16 @@ Content
       }),
     ]);
     expect(splits.length).toBe(4);
-    expect(splits[0].metadata).toEqual({ "Header 1": "Main Header" });
+    expect(splits[0].metadata).toEqual({ Header_1: "Main Header" });
     expect(splits[1].metadata).toEqual({
-      "Header 1": "Main Header",
-      "Header 2": "Sub-header",
+      Header_1: "Main Header",
+      Header_2: "Sub-header",
     });
     expect(splits[2].metadata).toEqual({
-      "Header 1": "Main Header",
-      "Header 2": "Sub-header",
-      "Header 3": "Sub-sub header",
+      Header_1: "Main Header",
+      Header_2: "Sub-header",
+      Header_3: "Sub-sub header",
     });
-    expect(splits[3].metadata).toEqual({ "Header 1": "New title" });
+    expect(splits[3].metadata).toEqual({ Header_1: "New title" });
   });
 });

packages/core/tests/node-parser/sentence-spiller.test.ts

@@ -1,12 +1,13 @@
+import { SentenceSplitter } from "@llamaindex/core/node-parser";
 import { Document } from "@llamaindex/core/schema";
-import { SimpleNodeParser } from "llamaindex/nodeParsers/index";
+import { tokenizers } from "@llamaindex/env";
 import { beforeEach, describe, expect, test } from "vitest";
 
-describe("SimpleNodeParser", () => {
-  let simpleNodeParser: SimpleNodeParser;
+describe("SentenceSplitter", () => {
+  let sentenceSplitter: SentenceSplitter;
 
   beforeEach(() => {
-    simpleNodeParser = new SimpleNodeParser({
+    sentenceSplitter = new SentenceSplitter({
       chunkSize: 1024,
       chunkOverlap: 20,
     });
@@ -19,7 +20,7 @@ describe("SimpleNodeParser", () => {
       excludedLlmMetadataKeys: ["animals"],
       excludedEmbedMetadataKeys: ["animals"],
     });
-    const result = simpleNodeParser.getNodesFromDocuments([doc]);
+    const result = sentenceSplitter.getNodesFromDocuments([doc]);
     expect(result.length).toEqual(1);
     const node = result[0];
     // check not the same object
@@ -37,4 +38,15 @@ describe("SimpleNodeParser", () => {
     // check relationship
     expect(node.sourceNode?.nodeId).toBe(doc.id_);
   });
+
+  test("split long text", async () => {
+    const longSentence = "is ".repeat(9000) + ".";
+    const document = new Document({ text: longSentence, id_: "1" });
+    const result = sentenceSplitter.getNodesFromDocuments([document]);
+    expect(result.length).toEqual(9);
+    result.forEach((node) => {
+      const { length } = tokenizers.tokenizer().encode(node.text);
+      expect(length).toBeLessThanOrEqual(1024);
+    });
+  });
 });

packages/core/tests/node-parser/sentence-window.test.ts

@@ -1,8 +1,5 @@
+import { SentenceWindowNodeParser } from "@llamaindex/core/node-parser";
 import { Document, MetadataMode } from "@llamaindex/core/schema";
-import {
-  DEFAULT_WINDOW_METADATA_KEY,
-  SentenceWindowNodeParser,
-} from "llamaindex/nodeParsers/index";
 import { describe, expect, test } from "vitest";
 
 describe("Tests for the SentenceWindowNodeParser class", () => {
@@ -11,7 +8,7 @@ describe("Tests for the SentenceWindowNodeParser class", () => {
     expect(sentenceWindowNodeParser).toBeDefined();
   });
   test("testing the getNodesFromDocuments method", () => {
-    const sentenceWindowNodeParser = SentenceWindowNodeParser.fromDefaults({
+    const sentenceWindowNodeParser = new SentenceWindowNodeParser({
       windowSize: 1,
     });
     const doc = new Document({ text: "Hello. Cat Mouse. Dog." });
@@ -25,7 +22,9 @@ describe("Tests for the SentenceWindowNodeParser class", () => {
       "Dog.",
     ]);
     expect(
-      resultingNodes.map((n) => n.metadata[DEFAULT_WINDOW_METADATA_KEY]),
+      resultingNodes.map(
+        (n) => n.metadata[SentenceWindowNodeParser.DEFAULT_WINDOW_METADATA_KEY],
+      ),
     ).toEqual([
       "Hello. Cat Mouse.",
       "Hello. Cat Mouse. Dog.",

packages/core/tests/node-parser/text-splitter.test.ts

@@ -1,23 +1,25 @@
 import {
   SentenceSplitter,
-  cjkSentenceTokenizer,
-} from "llamaindex/TextSplitter";
+  splitBySentenceTokenizer,
+} from "@llamaindex/core/node-parser";
 import { describe, expect, test } from "vitest";
 
-describe("SentenceSplitter", () => {
+describe("sentence splitter", () => {
   test("initializes", () => {
     const sentenceSplitter = new SentenceSplitter();
     expect(sentenceSplitter).toBeDefined();
   });
 
+  test("chunk size should less than chunk", async () => {});
+
   test("splits paragraphs w/o effective chunk size", () => {
     const sentenceSplitter = new SentenceSplitter({
-      paragraphSeparator: "\n\n\n",
+      chunkSize: 9,
+      chunkOverlap: 0,
     });
     // generate the same line as above but correct syntax errors
-    const splits = sentenceSplitter.getParagraphSplits(
+    const splits = sentenceSplitter.splitText(
       "This is a paragraph.\n\n\nThis is another paragraph.",
-      undefined,
     );
     expect(splits).toEqual([
       "This is a paragraph.",
@@ -30,9 +32,8 @@ describe("SentenceSplitter", () => {
       paragraphSeparator: "\n",
     });
     // generate the same line as above but correct syntax errors
-    const splits = sentenceSplitter.getParagraphSplits(
+    const splits = sentenceSplitter.splitText(
       "This is a paragraph.\nThis is another paragraph.",
-      1000,
     );
     expect(splits).toEqual([
       "This is a paragraph.\nThis is another paragraph.",
@@ -40,10 +41,12 @@ describe("SentenceSplitter", () => {
   });
 
   test("splits sentences", () => {
-    const sentenceSplitter = new SentenceSplitter();
-    const splits = sentenceSplitter.getSentenceSplits(
+    const sentenceSplitter = new SentenceSplitter({
+      chunkSize: 9,
+      chunkOverlap: 0,
+    });
+    const splits = sentenceSplitter.splitText(
       "This is a sentence. This is another sentence.",
-      undefined,
     );
     expect(splits).toEqual([
       "This is a sentence.",
@@ -89,9 +92,10 @@ describe("SentenceSplitter", () => {
 
   test("splits cjk", () => {
     const sentenceSplitter = new SentenceSplitter({
-      chunkSize: 12,
+      chunkSize: 30,
       chunkOverlap: 0,
-      chunkingTokenizerFn: cjkSentenceTokenizer,
+      secondaryChunkingRegex:
+        '.*?([﹒﹔﹖﹗．；。！？]["’”」』]{0,2}|：(?=["‘“「『]{1,2}|$))',
     });
 
     const splits = sentenceSplitter.splitText(
@@ -104,4 +108,11 @@ describe("SentenceSplitter", () => {
       "因为他照了人类，连我都在内。",
     ]);
   });
+
+  test("issue 1087 - edge case when input with brackets", () => {
+    const text =
+      "A card must be of uniform thickness and made of unfolded and uncreased paper or cardstock of approximately the quality and weight of a stamped card (i.e., a card available from USPS).";
+    const split = splitBySentenceTokenizer();
+    expect(split(text)).toEqual([text]);
+  });
 });

packages/core/tests/package.json

@@ -7,6 +7,6 @@
   },
   "devDependencies": {
     "@llamaindex/core": "workspace:*",
-    "vitest": "^1.6.0"
+    "vitest": "^2.0.2"
   }
 }

packages/env/CHANGELOG.md

@@ -1,5 +1,11 @@
 # @llamaindex/env
 
+## 0.1.9
+
+### Patch Changes
+
+- 58abc57: fix: align version
+
 ## 0.1.8
 
 ### Patch Changes

packages/env/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@llamaindex/env",
   "description": "environment wrapper, supports all JS environment including node, deno, bun, edge runtime, and cloudflare worker",
-  "version": "0.1.8",
+  "version": "0.1.9",
   "type": "module",
   "types": "dist/type/index.d.ts",
   "main": "dist/cjs/index.js",
@@ -68,11 +68,11 @@
   },
   "devDependencies": {
     "@aws-crypto/sha256-js": "^5.2.0",
-    "@swc/cli": "^0.3.12",
+    "@swc/cli": "^0.4.0",
     "@swc/core": "^1.6.3",
     "concurrently": "^8.2.2",
     "pathe": "^1.1.2",
-    "vitest": "^1.6.0"
+    "vitest": "^2.0.2"
   },
   "dependencies": {
     "@types/lodash": "^4.17.5",

packages/experimental/CHANGELOG.md

@@ -1,5 +1,115 @@
 # @llamaindex/experimental
 
+## 0.0.66
+
+### Patch Changes
+
+- 58abc57: fix: align version
+- Updated dependencies [58abc57]
+  - llamaindex@0.5.16
+
+## 0.0.65
+
+### Patch Changes
+
+- Updated dependencies [01c184c]
+- Updated dependencies [07a275f]
+  - llamaindex@0.5.15
+
+## 0.0.64
+
+### Patch Changes
+
+- Updated dependencies [c825a2f]
+  - llamaindex@0.5.14
+
+## 0.0.63
+
+### Patch Changes
+
+- llamaindex@0.5.13
+
+## 0.0.62
+
+### Patch Changes
+
+- Updated dependencies [345300f]
+- Updated dependencies [da5cfc4]
+- Updated dependencies [da5cfc4]
+  - llamaindex@0.5.12
+
+## 0.0.61
+
+### Patch Changes
+
+- llamaindex@0.5.11
+
+## 0.0.60
+
+### Patch Changes
+
+- Updated dependencies [086b940]
+- Updated dependencies [5d5716b]
+- Updated dependencies [91d02a4]
+- Updated dependencies [fb6db45]
+  - llamaindex@0.5.10
+
+## 0.0.59
+
+### Patch Changes
+
+- Updated dependencies [15962b3]
+  - llamaindex@0.5.9
+
+## 0.0.58
+
+### Patch Changes
+
+- Updated dependencies [3d5ba08]
+- Updated dependencies [d917cdc]
+  - llamaindex@0.5.8
+
+## 0.0.57
+
+### Patch Changes
+
+- Updated dependencies [ec59acd]
+  - llamaindex@0.5.7
+
+## 0.0.56
+
+### Patch Changes
+
+- Updated dependencies [2562244]
+- Updated dependencies [325aa51]
+- Updated dependencies [ab700ea]
+- Updated dependencies [92f0782]
+- Updated dependencies [6cf6ae6]
+- Updated dependencies [b7cfe5b]
+  - llamaindex@0.5.6
+
+## 0.0.55
+
+### Patch Changes
+
+- Updated dependencies [b974eea]
+  - llamaindex@0.5.5
+
+## 0.0.54
+
+### Patch Changes
+
+- Updated dependencies [1a65ead]
+  - llamaindex@0.5.4
+
+## 0.0.53
+
+### Patch Changes
+
+- Updated dependencies [9bbbc67]
+- Updated dependencies [b3681bf]
+  - llamaindex@0.5.3
+
 ## 0.0.52
 
 ### Patch Changes

packages/experimental/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@llamaindex/experimental",
   "description": "Experimental package for LlamaIndexTS",
-  "version": "0.0.52",
+  "version": "0.0.66",
   "type": "module",
   "types": "dist/type/index.d.ts",
   "main": "dist/cjs/index.js",
@@ -56,7 +56,7 @@
   },
   "devDependencies": {
     "@aws-crypto/sha256-js": "^5.2.0",
-    "@swc/cli": "^0.3.12",
+    "@swc/cli": "^0.4.0",
     "@swc/core": "^1.6.3",
     "@types/jsonpath": "^0.2.4",
     "concurrently": "^8.2.2",

packages/experimental/src/engines/query/JSONQueryEngine.ts

@@ -162,18 +162,18 @@ export class JSONQueryEngine implements QueryEngine {
 
     const schema = this.getSchemaContext();
 
-    const jsonPathResponseStr = await this.serviceContext.llm.complete({
+    const { text: jsonPathResponse } = await this.serviceContext.llm.complete({
       prompt: this.jsonPathPrompt({ query, schema }),
     });
 
     if (this.verbose) {
       console.log(
-        `> JSONPath Instructions:\n\`\`\`\n${jsonPathResponseStr}\n\`\`\`\n`,
+        `> JSONPath Instructions:\n\`\`\`\n${jsonPathResponse}\n\`\`\`\n`,
       );
     }
 
     const jsonPathOutput = await this.outputProcessor({
-      llmOutput: jsonPathResponseStr.text,
+      llmOutput: jsonPathResponse,
       jsonValue: this.jsonValue,
     });
 
@@ -188,7 +188,7 @@ export class JSONQueryEngine implements QueryEngine {
         prompt: this.responseSynthesisPrompt({
           query,
           jsonSchema: schema,
-          jsonPath: jsonPathResponseStr.text,
+          jsonPath: jsonPathResponse,
           jsonPathValue: JSON.stringify(jsonPathOutput),
         }),
       });
@@ -199,7 +199,7 @@ export class JSONQueryEngine implements QueryEngine {
     }
 
     const responseMetadata = {
-      jsonPathResponseStr,
+      jsonPathResponse,
     };
 
     const response = EngineResponse.fromResponse(responseStr, false);

packages/llamaindex/CHANGELOG.md

@@ -1,5 +1,126 @@
 # llamaindex
 
+## 0.5.16
+
+### Patch Changes
+
+- 58abc57: fix: align version
+- Updated dependencies [58abc57]
+  - @llamaindex/cloud@0.2.2
+  - @llamaindex/core@0.1.8
+  - @llamaindex/env@0.1.9
+
+## 0.5.15
+
+### Patch Changes
+
+- 01c184c: Add is_empty operator for filtering vector store
+- 07a275f: chore: bump openai
+
+## 0.5.14
+
+### Patch Changes
+
+- c825a2f: Add gpt-4o-mini to Azure. Add 2024-06-01 API version for Azure
+
+## 0.5.13
+
+### Patch Changes
+
+- Updated dependencies [04b2f8e]
+  - @llamaindex/core@0.1.7
+
+## 0.5.12
+
+### Patch Changes
+
+- 345300f: feat: add splitByPage mode to LlamaParseReader
+- da5cfc4: Add metadatafilter options to retriever constructors
+- da5cfc4: Fix system prompt not used in ContextChatEngine
+- Updated dependencies [0452af9]
+  - @llamaindex/core@0.1.6
+
+## 0.5.11
+
+### Patch Changes
+
+- Updated dependencies [1f680d7]
+  - @llamaindex/cloud@0.2.1
+
+## 0.5.10
+
+### Patch Changes
+
+- 086b940: feat: add DeepSeek LLM
+- 5d5716b: feat: add a reader for JSON data
+- 91d02a4: feat: support transform component callable
+- fb6db45: feat: add pageSeparator params to LlamaParseReader
+- Updated dependencies [91d02a4]
+  - @llamaindex/core@0.1.5
+
+## 0.5.9
+
+### Patch Changes
+
+- 15962b3: feat: node parser refactor
+
+  Align the text splitter logic with Python; it has almost the same logic as Python; Zod checks for input and better error messages and event system.
+
+  This change will not be considered a breaking change since it doesn't have a significant output difference from the last version,
+  but some edge cases will change, like the page separator and parameter for the constructor.
+
+- Updated dependencies [15962b3]
+  - @llamaindex/core@0.1.4
+
+## 0.5.8
+
+### Patch Changes
+
+- 3d5ba08: fix: update user agent in AssemblyAI
+- d917cdc: Add azure interpreter tool to tool factory
+
+## 0.5.7
+
+### Patch Changes
+
+- ec59acd: fix: bundling issue with pnpm
+
+## 0.5.6
+
+### Patch Changes
+
+- 2562244: feat: add gpt4o-mini
+- 325aa51: Implement Jina embedding through Jina api
+- ab700ea: Add missing authentication to LlamaCloudIndex.fromDocuments
+- 92f0782: feat: use query bundle
+- 6cf6ae6: feat: abstract query type
+- b7cfe5b: fix: passing max_token option to replicate's api call
+- Updated dependencies [6cf6ae6]
+  - @llamaindex/core@0.1.3
+
+## 0.5.5
+
+### Patch Changes
+
+- b974eea: Add support for Metadata filters
+- Updated dependencies [b974eea]
+  - @llamaindex/core@0.1.2
+
+## 0.5.4
+
+### Patch Changes
+
+- 1a65ead: feat: add vendorMultimodal params to LlamaParseReader
+
+## 0.5.3
+
+### Patch Changes
+
+- 9bbbc67: feat: add a reader for Discord messages
+- b3681bf: fix: DataCloneError when using FunctionTool
+- Updated dependencies [b3681bf]
+  - @llamaindex/core@0.1.1
+
 ## 0.5.2
 
 ### Patch Changes