`@xenova/transformers` only ship node.js and browser output, it's not possible to load this in edge runtime and workerd
This reverts commit 34fb1d8992.
* refactor: merge HistoryChatEngine and ContextChatEngine and use ChatHistory for all chat engines
* fix: add safeguard for tokensToSummarize
* refactor: unfold chat engines to own folder
* refactor: extract LLM types
* refactor: move multi-modal types to llm
* docs(changeset): Remove HistoryChatEngine and use ChatHistory for all chat engines
* dev: add debug launcher and don't lint generated code
description:Write something like "Modify the ... api endpoint to use ... version and ... framework"
labels:sweep
title:""
description:Write something like "Modify the ... api endpoint to use ... version and ... framework" If you would like to use sweep.dev prefix with "Sweep:"
body:
- type:textarea
id:description
attributes:
label:Details
description:More details for Sweep
description:More details
placeholder:We are migrating this function to ... version because ...
We recommend you to understand the basics of Node.js, TypeScript, pnpm, and of course, LLM before contributing.
packages/core which is the main NPM library llamaindex
There are some important folders in the repository:
apps/simple is where the demo code lives
### Turborepo docs
You can checkout how Turborepo works using the default [README-turborepo.md](/README-turborepo.md)
-`packages/*`: Contains the source code of the packages. Each package is a separate npm package.
-`llamaindex`: The starter package for LlamaIndex.TS, which contains the all sub-packages.
-`core`: The core package of LlamaIndex.TS, which contains the abstract classes and interfaces. It is designed for
all JS runtime environments.
-`env`: The environment package of LlamaIndex.TS, which contains the environment-specific classes and interfaces. It
includes compatibility layers for Node.js, Deno, Vercel Edge Runtime, Cloudflare Workers...
-`apps/*`: The applications based on LlamaIndex.TS.
-`next`: Our documentation website based on Next.js.
-`examples`: The code examples of LlamaIndex.TS using Node.js.
## Getting Started
Install NodeJS. Preferably v18 using nvm or n.
Inside the LlamaIndexTS directory:
Make sure you have Node.js LIS (Long-term Support) installed. You can check your Node.js version by running:
```shell
node -v
# v20.x.x
```
npm i -g pnpm ts-node
### Use pnpm
```shell
corepack enable
```
### Install dependencies
```shell
pnpm install
```
Note: we use pnpm in this repo, which has a lot of the same functionality and CLI options as npm but it does do some things better in a monorepo, like centralizing dependencies and caching.
### Build the packages
PNPM's has documentation on its [workspace feature](https://pnpm.io/workspaces) and Turborepo had some [useful documentation also](https://turbo.build/repo/docs/core-concepts/monorepos/running-tasks).
### Running Typescript
When we publish to NPM we will have a tsc compiled version of the library in JS. For now, the easiest thing to do is use ts-node.
### Test cases
To run them, run
```
pnpm run test
```
To write new test cases write them in [packages/core/src/tests](/packages/core/src/tests)
We use Jest https://jestjs.io/ to write our test cases. Jest comes with a bunch of built in assertions using the expect function: https://jestjs.io/docs/expect
### Demo applications
There is an existing ["simple"](/apps/simple/README.md) demos folder with mainly NodeJS scripts. Feel free to add additional demos to that folder. If you would like to try out your changes in the core package with a new demo, you need to run the build command in the README.
You can create new demo applications in the apps folder. Just run pnpm init in the folder after you create it to create its own package.json
### Installing packages
To install packages for a specific package or demo application, run
```
pnpm add [NPM Package] --filter [package or application i.e. core or simple]
```
To install packages for every package or application run
```
pnpm add -w [NPM Package]
```shell
# Build all packages
turbo build --filter "./packages/*"
```
### Docs
To contribute to the docs, go to the docs website folder and run the Docusaurus instance.
See the [docs](./apps/next/README.md) for more information.
```bash
cd apps/docs
pnpm install
pnpm start
## Changeset
We use [changesets](https://github.com/changesets/changesets) for managing versions and changelogs. To create a new
changeset, run in the root folder:
```
pnpm changeset
```
That should start a webserver which will serve the docs on https://localhost:3000
Please send a descriptive changeset for each PR.
Any changes you make should be reflected in the browser. If you need to regenerate the API docs and find that your TSDoc isn't getting the updates, feel free to remove apps/docs/api. It will automatically regenerate itself when you run pnpm start again.
## Publishing (maintainers only)
The [Release Github Action](.github/workflows/release.yml) is automatically generating and updating a
PR called "Release {version}".
This PR will update the `package.json` and `CHANGELOG.md` files of each package according to
the current changesets in the [.changeset](.changeset) folder.
If this PR is merged it will automatically add version tags to the repository and publish the updated packages to NPM.
This Turborepo includes the following packages/apps:
### Apps and Packages
-`docs`: a [Next.js](https://nextjs.org/) app
-`web`: another [Next.js](https://nextjs.org/) app
-`ui`: a stub React component library shared by both `web` and `docs` applications
-`eslint-config-custom`: `eslint` configurations (includes `eslint-config-next` and `eslint-config-prettier`)
-`tsconfig`: `tsconfig.json`s used throughout the monorepo
Each package/app is 100% [TypeScript](https://www.typescriptlang.org/).
### Utilities
This Turborepo has some additional tools already setup for you:
- [TypeScript](https://www.typescriptlang.org/) for static type checking
- [ESLint](https://eslint.org/) for code linting
- [Prettier](https://prettier.io) for code formatting
### Build
To build all apps and packages, run the following command:
```
cd my-turborepo
pnpm build
```
### Develop
To develop all apps and packages, run the following command:
```
cd my-turborepo
pnpm dev
```
### Remote Caching
Turborepo can use a technique known as [Remote Caching](https://turbo.build/repo/docs/core-concepts/remote-caching) to share cache artifacts across machines, enabling you to share build caches with your team and CI/CD pipelines.
By default, Turborepo will cache locally. To enable Remote Caching you will need an account with Vercel. If you don't have an account you can [create one](https://vercel.com/signup), then enter the following commands:
```
cd my-turborepo
npx turbo login
```
This will authenticate the Turborepo CLI with your [Vercel account](https://vercel.com/docs/concepts/personal-accounts/overview).
Next, you can link your Turborepo to your Remote Cache by running the following command from the root of your Turborepo:
LlamaIndex is a data framework for your LLM application.
Use your own data with large language models (LLMs, OpenAI ChatGPT and others) in Typescript and Javascript.
Use your own data with large language models (LLMs, OpenAI ChatGPT and others) in JS runtime environments with TypeScript support.
Documentation: https://ts.llamaindex.ai/
Try examples online:
[](https://stackblitz.com/github/run-llama/LlamaIndexTS/tree/main/examples)
## What is LlamaIndex.TS?
LlamaIndex.TS aims to be a lightweight, easy to use set of libraries to help you integrate large language models into your applications with your own data.
## Getting started with an example:
## Compatibility
LlamaIndex.TS requries Node v18 or higher. You can download it from https://nodejs.org or use https://nvm.sh (our preferred option).
The reason is that some classes are only compatible with Node.js runtime,(e.g. `PDFReader`) which uses Node.js specific APIs(like `fs`, `child_process`, `crypto`).
// Query the index
constqueryEngine=index.asQueryEngine();
constresponse=awaitqueryEngine.query(
"What did the author do in college?",
);
If you need any of those classes, you have to import them instead directly though their file path in the package.
Here's an example for importing the `PineconeVectorStore` class:
> _Note_: Reader classes have to be added explictly to the `fileExtToReader` map in the Edge version of the `SimpleDirectoryReader`.
```bash
pnpx ts-node example.ts
```
You'll find a complete example with LlamaIndexTS here: https://github.com/run-llama/create_llama_projects/tree/main/nextjs-edge-llamaparse
## Playground
@@ -70,50 +105,25 @@ Check out our NextJS playground at https://llama-playground.vercel.app/. The sou
## Core concepts for getting started:
- [Document](/packages/core/src/Node.ts): A document represents a text file, PDF file or other contiguous piece of data.
- [Document](/packages/llamaindex/src/Node.ts): A document represents a text file, PDF file or other contiguous piece of data.
- [Node](/packages/core/src/Node.ts): The basic data building block. Most commonly, these are parts of the document split into manageable pieces that are small enough to be fed into an embedding model and LLM.
- [Node](/packages/llamaindex/src/Node.ts): The basic data building block. Most commonly, these are parts of the document split into manageable pieces that are small enough to be fed into an embedding model and LLM.
- [Embedding](/packages/core/src/Embedding.ts): Embeddings are sets of floating point numbers which represent the data in a Node. By comparing the similarity of embeddings, we can derive an understanding of the similarity of two pieces of data. One use case is to compare the embedding of a question with the embeddings of our Nodes to see which Nodes may contain the data needed to answer that quesiton.
- [Embedding](/packages/llamaindex/src/embeddings/OpenAIEmbedding.ts): Embeddings are sets of floating point numbers which represent the data in a Node. By comparing the similarity of embeddings, we can derive an understanding of the similarity of two pieces of data. One use case is to compare the embedding of a question with the embeddings of our Nodes to see which Nodes may contain the data needed to answer that question. Because the default service context is OpenAI, the default embedding is `OpenAIEmbedding`. If using different models, say through Ollama, use this [Embedding](/packages/llamaindex/src/embeddings/OllamaEmbedding.ts) (see all [here](/packages/llamaindex/src/embeddings)).
- [Indices](/packages/core/src/indices/): Indices store the Nodes and the embeddings of those nodes. QueryEngines retrieve Nodes from these Indices using embedding similarity.
- [Indices](/packages/llamaindex/src/indices/): Indices store the Nodes and the embeddings of those nodes. QueryEngines retrieve Nodes from these Indices using embedding similarity.
- [QueryEngine](/packages/core/src/QueryEngine.ts): Query engines are what generate the query you put in and give you back the result. Query engines generally combine a pre-built prompt with selected Nodes from your Index to give the LLM the context it needs to answer your query.
- [QueryEngine](/packages/llamaindex/src/engines/query/RetrieverQueryEngine.ts): Query engines are what generate the query you put in and give you back the result. Query engines generally combine a pre-built prompt with selected Nodes from your Index to give the LLM the context it needs to answer your query. To build a query engine from your Index (recommended), use the [`asQueryEngine`](/packages/llamaindex/src/indices/BaseIndex.ts) method on your Index. See all query engines [here](/packages/llamaindex/src/engines/query).
- [ChatEngine](/packages/core/src/ChatEngine.ts): A ChatEngine helps you build a chatbot that will interact with your Indices.
- [ChatEngine](/packages/llamaindex/src/engines/chat/SimpleChatEngine.ts): A ChatEngine helps you build a chatbot that will interact with your Indices. See all chat engines [here](/packages/llamaindex/src/engines/chat).
- [SimplePrompt](/packages/core/src/Prompt.ts): A simple standardized function call definition that takes in inputs and formats them in a template literal. SimplePrompts can be specialized using currying and combined using other SimplePrompt functions.
## Note: NextJS:
If you're using NextJS App Router, you'll need to use the NodeJS runtime (default) and add the follow config to your next.config.js to have it use imports/exports in the same way Node does.
```js
exportconstruntime="nodejs";// default
```
```js
// next.config.js
/** @type {import('next').NextConfig} */
constnextConfig={
experimental:{
serverComponentsExternalPackages:["pdf-parse"],// Puts pdf-parse in actual NodeJS mode with NextJS App Router
},
};
module.exports=nextConfig;
```
## Supported LLMs:
- OpenAI GPT-3.5-turbo and GPT-4
- Anthropic Claude Instant and Claude 2
- Llama2 Chat LLMs (70B, 13B, and 7B parameters)
- [SimplePrompt](/packages/llamaindex/src/Prompt.ts): A simple standardized function call definition that takes in inputs and formats them in a template literal. SimplePrompts can be specialized using currying and combined using other SimplePrompt functions.
## Contributing:
We are in the very early days of LlamaIndex.TS. If you’re interested in hacking on it with us check out our [contributing guide](/CONTRIBUTING.md)
Please see our [contributing guide](CONTRIBUTING.md) for more information.
You are highly encouraged to contribute to LlamaIndex.TS!
We include several end-to-end examples using LlamaIndex.TS in the repository
Check out the examples below or try them out and complete them in minutes with interactive Github Codespace tutorials provided by Dev-Docs [here](https://codespaces.new/team-dev-docs/lits-dev-docs-playground?devcontainer_path=.devcontainer%2Fjavascript_ltsquickstart%2Fdevcontainer.json):
Create a list index and query it. This example also use the `LLMRetriever`, which will use the LLM to select the best nodes to use when generating answer.
## [Save / Load an Index](https://github.com/run-llama/LlamaIndexTS/blob/main/apps/simple/storageContext.ts)
Create and load a vector index. Persistance to disk in LlamaIndex.TS happens automatically once a storage context object is created.
Uses the `SubQuestionQueryEngine`, which breaks complex queries into multiple questions, and then aggreates a response across the answers to all sub-questions.
This example uses several low-level components, which removes the need for an actual query engine. These components can be used anywhere, in any application, or customized and sub-classed to meet your own needs.
import CodeSource from "!raw-loader!../../../../examples/chatEngine";
# Chat Engine
Chat Engine is a class that allows you to create a chatbot from a retriever. It is a wrapper around a retriever that allows you to chat with it in a conversational manner.
The Context-Aware Agent enhances the capabilities of standard LLM agents by incorporating relevant context from a retriever for each query. This allows the agent to provide more informed and specific responses based on the available information.
## Usage
Here's a simple example of how to use the Context-Aware Agent:
```typescript
import {
Document,
VectorStoreIndex,
OpenAIContextAwareAgent,
OpenAI,
} from "llamaindex";
async function createContextAwareAgent() {
// Create and index some documents
const documents = [
new Document({
text: "LlamaIndex is a data framework for LLM applications.",
id_: "doc1",
}),
new Document({
text: "The Eiffel Tower is located in Paris, France.",
id_: "doc2",
}),
];
const index = await VectorStoreIndex.fromDocuments(documents);
In this example, the Context-Aware Agent uses the retriever to fetch relevant context for each query, allowing it to provide more accurate and informed responses based on the indexed documents.
## Key Components
- `contextRetriever`: A retriever (e.g., from a VectorStoreIndex) that fetches relevant documents or passages for each query.
## Available Context-Aware Agents
- `OpenAIContextAwareAgent`: A context-aware agent using OpenAI's models.
- `AnthropicContextAwareAgent`: A context-aware agent using Anthropic's models.
LlamaIndex.TS supports OpenAI and [other remote LLM APIs](other_llms). You can also run a local LLM on your machine!
## Using a local model via Ollama
The easiest way to run a local LLM is via the great work of our friends at [Ollama](https://ollama.com/), who provide a simple to use client that will download, install and run a [growing range of models](https://ollama.com/library) for you.
### Install Ollama
They provide a one-click installer for Mac, Linux and Windows on their [home page](https://ollama.com/).
### Pick and run a model
Since we're going to be doing agentic work, we'll need a very capable model, but the largest models are hard to run on a laptop. We think `mixtral 8x7b` is a good balance between power and resources, but `llama3` is another great option. You can run Mixtral by running
```bash
ollama run mixtral:8x7b
```
The first time you run it will also automatically download and install the model for you.
### Switch the LLM in your code
To tell LlamaIndex to use a local LLM, use the `Settings` object:
```javascript
Settings.llm = new Ollama({
model: "mixtral:8x7b",
});
```
### Use local embeddings
If you're doing retrieval-augmented generation, LlamaIndex.TS will also call out to OpenAI to index and embed your data. To be entirely local, you can use a local embedding model like this:
```javascript
Settings.embedModel = new HuggingFaceEmbedding({
modelType: "BAAI/bge-small-en-v1.5",
quantized: false,
});
```
The first time this runs it will download the embedding model to run it.
### Try it out
With a local LLM and local embeddings in place, you can perform RAG as usual and everything will happen on your machine without calling an API:
Our GitHub repository has a wealth of examples to explore and try out. You can check out our [examples folder](https://github.com/run-llama/LlamaIndexTS/tree/main/examples) to see them all at once, or browse the pages in this section for some selected highlights.
## Check out all examples
It may be useful to check out all the examples at once so you can try them out locally. To do this into a folder called `my-new-project`, run these commands:
import CodeSource from "!raw-loader!../../../../examples/mistral";
# Using other LLM APIs
By default LlamaIndex.TS uses OpenAI's LLMs and embedding models, but we support [lots of other LLMs](../modules/llms) including models from Mistral (Mistral, Mixtral), Anthropic (Claude) and Google (Gemini).
If you don't want to use an API at all you can [run a local model](../../examples/local_llm)
## Using another LLM
You can specify what LLM LlamaIndex.TS will use on the `Settings` object, like this:
```typescript
import { MistralAI, Settings } from "llamaindex";
Settings.llm = new MistralAI({
model: "mistral-tiny",
apiKey: "<YOUR_API_KEY>",
});
```
You can see examples of other APIs we support by checking out "Available LLMs" in the sidebar of our [LLMs section](../modules/llms).
## Using another embedding model
A frequent gotcha when trying to use a different API as your LLM is that LlamaIndex will also by default index and embed your data using OpenAI's embeddings. To completely switch away from OpenAI you will need to set your embedding model as well, for example:
```typescript
import { MistralAIEmbedding, Settings } from "llamaindex";
Settings.embedModel = new MistralAIEmbedding();
```
We support [many different embeddings](../modules/embeddings).
## Full example
This example uses Mistral's `mistral-tiny` model as the LLM and Mistral for embeddings as well.
A data connector (i.e. `Reader`) ingest data from different data sources and data formats into a simple `Document` representation (text and simple metadata).
[**Documents / Nodes**](./modules/high_level/documents_and_nodes.md): A `Document` is a generic container around any data source - for instance, a PDF, an API output, or retrieved data from a database. A `Node` is the atomic unit of data in LlamaIndex and represents a "chunk" of a source `Document`. It's a rich representation that includes metadata and relationships (to other nodes) to enable accurate and expressive retrieval operations.
[**Documents / Nodes**](../modules/documents_and_nodes/index.md): A `Document` is a generic container around any data source - for instance, a PDF, an API output, or retrieved data from a database. A `Node` is the atomic unit of data in LlamaIndex and represents a "chunk" of a source `Document`. It's a rich representation that includes metadata and relationships (to other nodes) to enable accurate and expressive retrieval operations.
Once you've ingested your data, LlamaIndex helps you index data into a format that's easy to retrieve.
Under the hood, LlamaIndex parses the raw documents into intermediate representations, calculates vector embeddings, and stores your data in-memory or to disk.
@@ -56,23 +56,23 @@ LlamaIndex provides composable modules that help you build and integrate RAG pip
These building blocks can be customized to reflect ranking preferences, as well as composed to reason over multiple knowledge bases in a structured way.
We support Node.JS versions 18, 20 and 22, with experimental support for Deno, Bun and Vercel Edge functions.
## NextJS
If you're using NextJS you'll need to add `withLlamaIndex` to your `next.config.js` file. This will add the necessary configuration for included 3rd-party libraries to your build:
```js
// next.config.js
constwithLlamaIndex=require("llamaindex/next");
module.exports=withLlamaIndex({
// your next.js config
});
```
For details, check the latest [withLlamaIndex](https://github.com/run-llama/LlamaIndexTS/blob/main/packages/llamaindex/src/next.ts) implementation.
We support Node.JS versions 18, 20 and 22, with experimental support for Deno, Bun and Vercel Edge functions.
## Installation from NPM
```bash npm2yarn
npm install llamaindex
```
### Environment variables
Our examples use OpenAI by default. You can use [other LLMs](../examples/other_llms) via their APIs; if you would prefer to use local models check out our [local LLM example](../examples/local_llm).
To use OpenAI, you'll need to [get an OpenAI API key](https://platform.openai.com/account/api-keys) and then make it available as an environment variable this way:
```bash
export OPENAI_API_KEY="sk-......" # Replace with your key
```
If you want to have it automatically loaded every time, add it to your `.zshrc/.bashrc`.
**WARNING:** do not check in your OpenAI key into version control. GitHub automatically invalidates OpenAI keys checked in by accident.
## What next?
- The easiest way to started is to [build a full-stack chat app with `create-llama`](starter_tutorial/chatbot).
- Try our other [getting started tutorials](starter_tutorial/retrieval_augmented_generation)
- Learn more about the [high level concepts](concepts) behind how LlamaIndex works
- Check out our [many examples](../examples/more_examples) of LlamaIndex.TS in action
import CodeSource from "!raw-loader!../../../../../examples/agent/openai";
# Agent tutorial
We have a comprehensive, step-by-step [guide to building agents in LlamaIndex.TS](../../guides/agents/setup) that we recommend to learn what agents are and how to build them for production. But building a basic agent is simple:
## Set up
In a new folder:
```bash npm2yarn
npm init
npm install -D typescript @types/node
```
## Run agent
Create the file `example.ts`. This code will:
- Create two tools for use by the agent:
- A `sumNumbers` tool that adds two numbers
- A `divideNumbers` tool that divides numbers
-
- Give an example of the data structure we wish to generate
- Prompt the LLM with instructions and the example, plus a sample transcript
<CodeBlock language="ts">{CodeSource}</CodeBlock>
To run the code:
```bash
npx tsx example.ts
```
You should expect output something like:
```
{
content: 'The sum of 5 + 5 is 10. When you divide 10 by 2, you get 5.',
Once you've mastered basic [retrieval-augment generation](retrieval_augmented_generation) you may want to create an interface to chat with your data. You can do this step-by-step, but we recommend getting started quickly using `create-llama`.
## Using create-llama
`create-llama` is a powerful but easy to use command-line tool that generates a working, full-stack web application that allows you to chat with your data. You can learn more about it on [the `create-llama` README page](https://www.npmjs.com/package/create-llama).
Run it once and it will ask you a series of questions about the kind of application you want to generate. Then you can customize your application to suit your use-case. To get started, run:
```bash npm2yarn
npx create-llama@latest
```
Once your app is generated, `cd` into your app directory and run
```bash npm2yarn
npm run dev
```
to start the development server. You can then visit [http://localhost:3000](http://localhost:3000) to see your app, which should look something like this:
import CodeSource from "!raw-loader!../../../../../examples/vectorIndex";
import TSConfigSource from "!!raw-loader!../../../../../examples/tsconfig.json";
# Retrieval Augmented Generation (RAG) Tutorial
One of the most common use-cases for LlamaIndex is Retrieval-Augmented Generation or RAG, in which your data is indexed and selectively retrieved to be given to an LLM as source material for responding to a query. You can learn more about the [concepts behind RAG](../concepts).
## Set up the project
In a new folder, run:
```bash npm2yarn
npm init
npm install -D typescript @types/node
```
Then, check out the [installation](../installation) steps to install LlamaIndex.TS and prepare an OpenAI key.
You can use [other LLMs](../../examples/other_llms) via their APIs; if you would prefer to use local models check out our [local LLM example](../../examples/local_llm).
## Run queries
Create the file `example.ts`. This code will
- load an example file
- convert it into a Document object
- index it (which creates embeddings using OpenAI)
- create a query engine to answer questions about the data
In college, the author studied subjects like linear algebra and physics, but did not find them particularly interesting. They started slacking off, skipping lectures, and eventually stopped attending classes altogether. They also had a negative experience with their English classes, where they were required to pay for catch-up training despite getting verbal approval to skip most of the classes. Ultimately, the author lost motivation for college due to their job as a software developer and stopped attending classes, only returning years later to pick up their papers.
0: Score: 0.8305309270895813 - I started this decade as a first-year college stud...
1: Score: 0.8286388215713089 - A short digression. I’m not saying colleges are wo...
```
Once you've mastered basic RAG, you may want to consider [chatting with your data](chatbot).
import CodeSource from "!raw-loader!../../../../../examples/jsonExtract";
# Structured data extraction tutorial
Make sure you have installed LlamaIndex.TS and have an OpenAI key. If you haven't, check out the [installation](../installation) guide.
You can use [other LLMs](../../examples/other_llms) via their APIs; if you would prefer to use local models check out our [local LLM example](../../examples/local_llm).
## Set up
In a new folder:
```bash npm2yarn
npm init
npm install -D typescript @types/node
```
## Extract data
Create the file `example.ts`. This code will:
- Set up an LLM connection to GPT-4
- Give an example of the data structure we wish to generate
- Prompt the LLM with instructions and the example, plus a sample transcript
<CodeBlock language="ts">{CodeSource}</CodeBlock>
To run the code:
```bash
npx tsx example.ts
```
You should expect output something like:
```json
{
"summary": "Sarah from XYZ Company called John to introduce the XYZ Widget, a tool designed to automate tasks and improve productivity. John expressed interest and requested case studies and a product demo. Sarah agreed to send the information and follow up to schedule the demo.",
"products": ["XYZ Widget"],
"rep_name": "Sarah",
"prospect_name": "John",
"action_items": [
"Send case studies and additional product information to John",
In this guide we'll walk you through the process of building an Agent in JavaScript using the LlamaIndex.TS library, starting from nothing and adding complexity in stages.
## What is an Agent?
In LlamaIndex, an agent is a semi-autonomous piece of software powered by an LLM that is given a task and executes a series of steps towards solving that task. It is given a set of tools, which can be anything from arbitrary functions up to full LlamaIndex query engines, and it selects the best available tool to complete each step. When each step is completed, the agent judges whether the task is now complete, in which case it returns a result to the user, or whether it needs to take another step, in which case it loops back to the start.
## Install LlamaIndex.TS
You'll need to have a recent version of [Node.js](https://nodejs.org/en) installed. Then you can install LlamaIndex.TS by running
```bash
npm install llamaindex
```
## Choose your model
By default we'll be using OpenAI with GPT-4, as it's a powerful model and easy to get started with. If you'd prefer to run a local model, see [using a local model](local_model).
## Get an OpenAI API key
If you don't already have one, you can sign up for an [OpenAI API key](https://platform.openai.com/api-keys). You should then put the key in a `.env` file in the root of the project; the file should look like
```
OPENAI_API_KEY=sk-XXXXXXXXXXXXXXXXXXXXXXXX
```
We'll use `dotenv` to pull the API key out of that .env file, so also run:
```bash
npm install dotenv
```
Now you're ready to [create your agent](create_agent).
We want to use `await` so we're going to wrap all of our code in a `main` function, like this:
```typescript
// Your imports go here
async function main() {
// the rest of your code goes here
}
main().catch(console.error);
```
For the rest of this guide we'll assume your code is wrapped like this so we can use `await`. You can run the code this way:
```bash
npx tsx example.ts
```
### Load your dependencies
First we'll need to pull in our dependencies. These are:
- The OpenAI class to use the OpenAI LLM
- FunctionTool to provide tools to our agent
- OpenAIAgent to create the agent itself
- Settings to define some global settings for the library
- Dotenv to load our API key from the .env file
```javascript
import { OpenAI, FunctionTool, OpenAIAgent, Settings } from "llamaindex";
import "dotenv/config";
```
### Initialize your LLM
We need to tell our OpenAI class where its API key is, and which of OpenAI's models to use. We'll be using `gpt-4o`, which is capable while still being pretty cheap. This is a global setting, so anywhere an LLM is needed will use the same model.
```javascript
Settings.llm = new OpenAI({
apiKey: process.env.OPENAI_API_KEY,
model: "gpt-4o",
});
```
### Turn on logging
We want to see what our agent is up to, so we're going to hook into some events that the library generates and print them out. There are several events possible, but we'll specifically tune in to `llm-tool-call` (when a tool is called) and `llm-tool-result` (when it responds).
We're going to create a very simple function that adds two numbers together. This will be the tool we ask our agent to use.
```javascript
const sumNumbers = ({ a, b }) => {
return `${a + b}`;
};
```
Note that we're passing in an object with two named parameters, `a` and `b`. This is a little unusual, but important for defining a tool that an LLM can use.
### Turn the function into a tool for the agent
This is the most complicated part of creating an agent. We need to define a `FunctionTool`. We have to pass in:
- The function itself (`sumNumbers`)
- A name for the function, which the LLM will use to call it
- A description of the function. The LLM will read this description to figure out what the tool does, and if it needs to call it
- A schema for function. We tell the LLM that the parameter is an `object`, and we tell it about the two named parameters we gave it, `a` and `b`. We describe each parameter as a `number`, and we say that both are required.
- You can see [more examples of function schemas](https://cookbook.openai.com/examples/how_to_call_functions_with_chat_models).
```javascript
const tool = FunctionTool.from(sumNumbers, {
name: "sumNumbers",
description: "Use this function to sum two numbers",
parameters: {
type: "object",
properties: {
a: {
type: "number",
description: "First number to sum",
},
b: {
type: "number",
description: "Second number to sum",
},
},
required: ["a", "b"],
},
});
```
We then wrap up the tools into an array. We could provide lots of tools this way, but for this example we're just using the one.
```javascript
const tools = [tool];
```
### Create the agent
With your LLM already set up and your tools defined, creating an agent is simple:
```javascript
const agent = new OpenAIAgent({ tools });
```
### Ask the agent a question
We can use the `chat` interface to ask our agent a question, and it will use the tools we've defined to find an answer.
```javascript
let response = await agent.chat({
message: "Add 101 and 303",
});
console.log(response);
```
Let's see what running this looks like using `npx tsx agent.ts`
We're seeing two pieces of output here. The first is our callback firing when the tool is called. You can see in `toolResult` that the LLM has correctly passed `101` and `303` to our `sumNumbers` function, which adds them up and returns `404`.
The second piece of output is the response from the LLM itself, where the `message.content` key is giving us the answer.
Great! We've built an agent with tool use! Next you can:
- [See the full code](https://github.com/run-llama/ts-agents/blob/main/1_agent/agent.ts)
- [Switch to a local LLM](local_model)
- Move on to [add Retrieval-Augmented Generation to your agent](agentic_rag)
If you're happy using OpenAI, you can skip this section, but many people are interested in using models they run themselves. The easiest way to do this is via the great work of our friends at [Ollama](https://ollama.com/), who provide a simple to use client that will download, install and run a [growing range of models](https://ollama.com/library) for you.
### Install Ollama
They provide a one-click installer for Mac, Linux and Windows on their [home page](https://ollama.com/).
### Pick and run a model
Since we're going to be doing agentic work, we'll need a very capable model, but the largest models are hard to run on a laptop. We think `mixtral 8x7b` is a good balance between power and resources, but `llama3` is another great option. You can run it simply by running
```bash
ollama run mixtral:8x7b
```
The first time you run it will also automatically download and install the model for you.
### Switch the LLM in your code
There are two changes you need to make to the code we already wrote in `1_agent` to get Mixtral 8x7b to work. First, you need to switch to that model. Replace the call to `Settings.llm` with this:
```javascript
Settings.llm = new Ollama({
model: "mixtral:8x7b",
});
```
### Swap to a ReActAgent
In our original code we used a specific OpenAIAgent, so we'll need to switch to a more generic agent pattern, the ReAct pattern. This is simple: change the `const agent` line in your code to read
```javascript
const agent = new ReActAgent({ tools });
```
(You will also need to bring in `Ollama` and `ReActAgent` in your imports)
### Run your totally local agent
Because your embeddings were already local, your agent can now run entirely locally without making any API calls.
```bash
node agent.mjs
```
Note that your model will probably run a lot slower than OpenAI, so be prepared to wait a while!
**_Output_**
```javascript
{
response: {
message: {
role: 'assistant',
content: ' Thought: I need to use a tool to add the numbers 101 and 303.\n' +
'Action: sumNumbers\n' +
'Action Input: {"a": 101, "b": 303}\n' +
'\n' +
'Observation: 404\n' +
'\n' +
'Thought: I can answer without using any more tools.\n' +
'Answer: The sum of 101 and 303 is 404.'
},
raw: {
model: 'mixtral:8x7b',
created_at: '2024-05-09T00:24:30.339473Z',
message: [Object],
done: true,
total_duration: 64678371209,
load_duration: 57394551334,
prompt_eval_count: 475,
prompt_eval_duration: 4163981000,
eval_count: 94,
eval_duration: 3116692000
}
},
sources: [Getter]
}
```
Tada! You can see all of this in the folder `1a_mixtral`.
### Extending to other examples
You can use a ReActAgent instead of an OpenAIAgent in any of the further examples below, but keep in mind that GPT-4 is a lot more capable than Mixtral 8x7b, so you may see more errors or failures in reasoning if you are using an entirely local setup.
### Next steps
Now you've got a local agent, you can [add Retrieval-Augmented Generation to your agent](agentic_rag).
While an agent that can perform math is nifty (LLMs are usually not very good at math), LLM-based applications are always more interesting when they work with large amounts of data. In this case, we're going to use a 200-page PDF of the proposed budget of the city of San Francisco for fiscal years 2024-2024 and 2024-2025. It's a great example because it's extremely wordy and full of tables of figures, which present a challenge for humans and LLMs alike.
To learn more about RAG, we recommend this [introduction](https://docs.llamaindex.ai/en/stable/getting_started/concepts/) from our Python docs. We'll assume you know the basics:
- Parse your source data into chunks of text.
- Encode that text as numbers, called embeddings.
- Search your embeddings for the most relevant chunks of text.
- Use the relevant chunks along with a query to ask an LLM to generate an answer.
We're going to start with the same agent we [built in step 1](https://github.com/run-llama/ts-agents/blob/main/1_agent/agent.ts), but make a few changes. You can find the finished version [in the repository](https://github.com/run-llama/ts-agents/blob/main/2_agentic_rag/agent.ts).
### New dependencies
We'll be bringing in `SimpleDirectoryReader`, `HuggingFaceEmbedding`, `VectorStoreIndex`, and `QueryEngineTool`, `OpenAIContextAwareAgent` from LlamaIndex.TS, as well as the dependencies we previously used.
```javascript
import {
OpenAI,
FunctionTool,
OpenAIAgent,
OpenAIContextAwareAgent,
Settings,
SimpleDirectoryReader,
HuggingFaceEmbedding,
VectorStoreIndex,
QueryEngineTool,
} from "llamaindex";
```
### Add an embedding model
To encode our text into embeddings, we'll need an embedding model. We could use OpenAI for this but to save on API calls we're going to use a local embedding model from HuggingFace.
```javascript
Settings.embedModel = new HuggingFaceEmbedding({
modelType: "BAAI/bge-small-en-v1.5",
quantized: false,
});
```
### Load data using SimpleDirectoryReader
`SimpleDirectoryReader` is a flexible tool that can read various file formats. We will point it at our data directory, which contains a single PDF file, and retrieve a set of documents.
We will convert our text into embeddings using the `VectorStoreIndex` class through the `fromDocuments` method, which utilizes the embedding model defined earlier in `Settings`.
```javascript
const index = await VectorStoreIndex.fromDocuments(documents);
```
### Configure a retriever
Before LlamaIndex can send a query to the LLM, it needs to find the most relevant chunks to send. That's the purpose of a `Retriever`. We're going to get `VectorStoreIndex` to act as a retriever for us
```javascript
const retriever = await index.asRetriever();
```
### Configure how many documents to retrieve
By default LlamaIndex will retrieve just the 2 most relevant chunks of text. This document is complex though, so we'll ask for more context.
```javascript
retriever.similarityTopK = 10;
```
### Approach 1: Create a Context-Aware Agent
With the retriever ready, you can create a **context-aware agent**.
```javascript
const agent = new OpenAIContextAwareAgent({
contextRetriever: retriever,
});
// Example query to the context-aware agent
let response = await agent.chat({
message: `What's the budget of San Francisco in 2023-2024?`,
});
console.log(response);
```
**Expected Output:**
```md
The total budget for the City and County of San Francisco for the fiscal year 2023-2024 is $14.6 billion. This represents a $611.8 million, or 4.4 percent, increase over the previous fiscal year's budget. The budget covers various expenditures across different departments and services, including significant allocations to public works, transportation, commerce, public protection, and health services.
```
### Approach 2: Using QueryEngineTool (Alternative Approach)
If you prefer more flexibility and don't mind additional complexity, you can create a `QueryEngineTool`. This approach allows you to define the query logic, providing a more tailored way to interact with the data, but note that it introduces a delay due to the extra tool call.
description: `This tool can answer detailed questions about the individual components of the budget of San Francisco in 2023-2024.`,
},
}),
];
// Create an agent using the tools array
const agent = new OpenAIAgent({ tools });
let toolResponse = await agent.chat({
message: "What's the budget of San Francisco in 2023-2024?",
});
console.log(toolResponse);
```
**Expected Output:**
```javascript
{
toolCall: {
id: 'call_iNo6rTK4pOpOBbO8FanfWLI9',
name: 'san_francisco_budget_tool',
input: { query: 'total budget' }
},
toolResult: {
tool: QueryEngineTool {
queryEngine: [RetrieverQueryEngine],
metadata: [Object]
},
input: { query: 'total budget' },
output: 'The total budget for the City and County of San Francisco for Fiscal Year (FY) 2023-24 is $14.6 billion, which represents a $611.8 million, or 4.4 percent, increase over the FY 2022-23 budget. For FY 2024-25, the total budget is also projected to be $14.6 billion, reflecting a $40.5 million, or 0.3 percent, decrease from the FY 2023-24 proposed budget. This budget includes various expenditures across different departments and services, with significant allocations to public works, transportation, commerce, public protection, and health services.',
isError: false
}
}
```
Once again we see a `toolResult`. You can see the query the LLM decided to send to the query engine ("total budget"), and the output the engine returned. In `response.message` you see that the LLM has returned the output from the tool almost verbatim, although it trimmed out the bit about 2024-2025 since we didn't ask about that year.
### Comparison of Approaches
The `OpenAIContextAwareAgent` approach simplifies the setup by allowing you to directly link the retriever to the agent, making it straightforward to access relevant context for your queries. This is ideal for situations where you want easy integration with existing data sources, like a context chat engine.
On the other hand, using the `QueryEngineTool` offers more flexibility and power. This method allows for customization in how queries are constructed and executed, enabling you to query data from various storages and process them in different ways. However, this added flexibility comes with increased complexity and response time due to the separate tool call and queryEngine generating tool output by LLM that is then passed to the agent.
So now we have an agent that can index complicated documents and answer questions about them. Let's [combine our math agent and our RAG agent](rag_and_tools)!
In [our third iteration of the agent](https://github.com/run-llama/ts-agents/blob/main/3_rag_and_tools/agent.ts) we've combined the two previous agents, so we've defined both `sumNumbers` and a `QueryEngineTool` and created an array of two tools:
```javascript
// define the query engine as a tool
const tools = [
new QueryEngineTool({
queryEngine: queryEngine,
metadata: {
name: "san_francisco_budget_tool",
description: `This tool can answer detailed questions about the individual components of the budget of San Francisco in 2023-2024.`,
},
}),
FunctionTool.from(sumNumbers, {
name: "sumNumbers",
description: "Use this function to sum two numbers",
parameters: {
type: "object",
properties: {
a: {
type: "number",
description: "First number to sum",
},
b: {
type: "number",
description: "Second number to sum",
},
},
required: ["a", "b"],
},
}),
];
```
These tool descriptions are identical to the ones we previously defined. Now let's ask it 3 questions in a row:
```javascript
let response = await agent.chat({
message:
"What's the budget of San Francisco for community health in 2023-24?",
});
console.log(response);
let response2 = await agent.chat({
message:
"What's the budget of San Francisco for public protection in 2023-24?",
});
console.log(response2);
let response3 = await agent.chat({
message:
"What's the combined budget of San Francisco for community health and public protection in 2023-24?",
});
console.log(response3);
```
We'll abbreviate the output, but here are the important things to spot:
```javascript
{
toolCall: {
id: 'call_ZA1LPx03gO4ABre1r6XowLWq',
name: 'san_francisco_budget_tool',
input: { query: 'community health budget 2023-2024' }
},
toolResult: {
tool: QueryEngineTool {
queryEngine: [RetrieverQueryEngine],
metadata: [Object]
},
input: { query: 'community health budget 2023-2024' },
output: 'The proposed Fiscal Year (FY) 2023-24 budget for the Department of Public Health is $3.2 billion
}
}
```
This is the first tool call, where it used the query engine to get the public health budget.
Complicated PDFs can be very tricky for LLMs to understand. To help with this, LlamaIndex provides LlamaParse, a hosted service that parses complex documents including PDFs. To use it, get a `LLAMA_CLOUD_API_KEY` by [signing up for LlamaCloud](https://cloud.llamaindex.ai/) (it's free for up to 1000 pages/day) and adding it to your `.env` file just as you did for your OpenAI key:
```bash
LLAMA_CLOUD_API_KEY=llx-XXXXXXXXXXXXXXXX
```
Then replace `SimpleDirectoryReader` with `LlamaParseReader`:
```javascript
const reader = new LlamaParseReader({ resultType: "markdown" });
Now you will be able to ask more complicated questions of the same PDF and get better results. You can find this code [in our repo](https://github.com/run-llama/ts-agents/blob/main/4_llamaparse/agent.ts).
Next up, let's persist our embedded data so we don't have to re-parse every time by [using a vector store](qdrant).
In the previous examples, we've been loading our data into memory each time we run the agent. This is fine for small datasets, but for larger datasets you'll want to store your embeddings in a database. LlamaIndex.TS provides a `VectorStore` class that can store your embeddings in a variety of databases. We're going to use [Qdrant](https://qdrant.tech/), a popular vector store, for this example.
We can get a local instance of Qdrant running very simply with Docker (make sure you [install Docker](https://www.docker.com/products/docker-desktop/) first):
```bash
docker pull qdrant/qdrant
docker run -p 6333:6333 qdrant/qdrant
```
And in our code we initialize a `VectorStore` with the Qdrant URL:
```javascript
// initialize qdrant vector store
const vectorStore = new QdrantVectorStore({
url: "http://localhost:6333",
});
```
Now once we have loaded our documents, we can instantiate an index with the vector store:
```javascript
// create a query engine from our documents
const index = await VectorStoreIndex.fromDocuments(documents, { vectorStore });
```
In [the final iteration](https://github.com/run-llama/ts-agents/blob/main/5_qdrant/agent.ts) you can see that we have also implemented a very naive caching mechanism to avoid re-parsing the PDF each time we run the agent:
LlamaIndex.TS is a data framework for LLM applications to ingest, structure, and access private or domain-specific data. While a python package is also available (see [here](https://gpt-index.readthedocs.io/en/latest/)), LlamaIndex.TS offers core features in a simple package, optimized for usage with TypeScript.
LlamaIndex is a framework for building LLM-powered applications. LlamaIndex helps you ingest, structure, and access private or domain-specific data. It's available [as a Python package](https://docs.llamaindex.ai/en/stable/) and in TypeScript (this package). LlamaIndex.TS offers the core features of LlamaIndex for popular runtimes like Node.js (official support), Vercel Edge Functions (experimental), and Deno (experimental).
## 🚀 Why LlamaIndex.TS?
At their core, LLMs offer a natural language interface between humans and inferred data. Widely available models come pre-trained on huge amounts of publicly available data, from Wikipedia and mailing lists to textbooks and source code.
LLMs offer a natural language interface between humans and inferred data. Widely available models come pre-trained on huge amounts of publicly available data, from Wikipedia and mailing lists to textbooks and source code.
Applications built on top of LLMs often require augmenting these models with private or domain-specific data. Unfortunately, that data can be distributed across siloed applications and data stores. It's behind APIs, in SQL databases, or trapped in PDFs and slide decks.
Applications built on top of LLMs often require augmenting these models with private or domain-specific data. That data is often distributed across siloed applications and data stores. It's behind APIs, in SQL databases, or trapped in PDFs and slide decks.
That's where **LlamaIndex.TS** comes in.
LlamaIndex.TS helps you unlock that data and then build powerful applications with it.
## 🦙 How can LlamaIndex.TS help?
## 🦙 What is LlamaIndex for?
LlamaIndex.TS provides the following tools:
LlamaIndex.TS handles several major use cases:
- **Data loading** ingest your existing `.txt`, `.pdf`, `.csv`, `.md` and `.docx` data directly
- **Data indexes** structure your data in intermediate representations that are easy and performant for LLMs to consume.
- **Engines** provide natural language access to your data. For example:
- Query engines are powerful retrieval interfaces for knowledge-augmented output.
- Chat engines are conversational interfaces for multi-message, "back and forth" interactions with your data.
- **Structured Data Extraction**: turning complex, unstructured and semi-structured data into uniform, programmatically accessible formats.
- **Retrieval-Augmented Generation (RAG)**: answering queries across your internal data by providing LLMs with up-to-date, semantically relevant context including Question and Answer systems and chat bots.
- **Autonomous Agents**: building software that is capable of intelligently selecting and using tools to accomplish tasks in an interactive, unsupervised manner.
## 👨👩👧👦 Who is LlamaIndex for?
LlamaIndex.TS provides a core set of tools, essential for anyone building LLM apps with JavaScript and TypeScript.
LlamaIndex targets the "AI Engineer": developers building software in any domain that can be enhanced by LLM-powered functionality, without needing to be an expert in machine learning or natural language processing.
Our high-level API allows beginner users to use LlamaIndex.TS to ingest and query their data.
Our high-level API allows beginner users to use LlamaIndex.TS to ingest, index, and query their data in just a few lines of code.
For more complex applications, our lower-level APIs allow advanced users to customize and extend any module—data connectors, indices, retrievers, and query engines, to fit their needs.
@@ -37,9 +35,9 @@ For more complex applications, our lower-level APIs allow advanced users to cust
`npm install llamaindex`
Our documentation includes [Installation Instructions](./installation.md) and a [Starter Tutorial](./starter.md) to build your first application.
Our documentation includes [Installation Instructions](./getting_started/installation.mdx) and a [Starter Tutorial](./getting_started/starter_tutorial/retrieval_augmented_generation.mdx) to build your first application.
Once you're up and running, [High-Level Concepts](./concepts.md) has an overview of LlamaIndex's modular architecture. For more hands-on practical examples, look through our [End-to-End Tutorials](./end_to_end.md).
Once you're up and running, [High-Level Concepts](./getting_started/concepts.md) has an overview of LlamaIndex's modular architecture. For more hands-on practical examples, look through our Examples section on the sidebar.
An “agent” is an automated reasoning and decision engine. It takes in a user input/query and can make internal decisions for executing that query in order to return the correct result. The key agent components can include, but are not limited to:
- Breaking down a complex question into smaller ones
- Choosing an external Tool to use + coming up with parameters for calling the Tool
- Planning out a set of tasks
- Storing previously completed tasks in a memory module
## Getting Started
LlamaIndex.TS comes with a few built-in agents, but you can also create your own. The built-in agents include:
- OpenAI Agent
- Anthropic Agent both via Anthropic and Bedrock (in `@llamaIndex/community`)
- Gemini Agent
- ReACT Agent
- Meta3.1 504B via Bedrock (in `@llamaIndex/community`)
import CodeSource from "!raw-loader!../../../../../examples/readers/src/discord";
# DiscordReader
DiscordReader is a simple data loader that reads all messages in a given Discord channel and returns them as Document objects.
It uses the [@discordjs/rest](https://github.com/discordjs/discord.js/tree/main/packages/rest) library to fetch the messages.
## Usage
First step is to create a Discord Application and generating a bot token [here](https://discord.com/developers/applications).
In your Discord Application, go to the `OAuth2` tab and generate an invite URL by selecting `bot` and click `Read Messages/View Channels` as wells as `Read Message History`.
This will invite the bot with the necessary permissions to read messages.
Copy the URL in your browser and select the server you want your bot to join.
<CodeBlock language="ts">{CodeSource}</CodeBlock>
### Params
#### DiscordReader()
- `discordToken?`: The Discord bot token.
- `requestHandler?`: Optionally provide a custom request function for edge environments, e.g. `fetch`. See discord.js for more info.
#### DiscordReader.loadData
- `channelIDs`: The ID(s) of discord channels as an array of strings.
- `limit?`: Optionally limit the number of messages to read
- `additionalInfo?`: An optional flag to include embedded messages and attachment urls in the document.
- `oldestFirst?`: An optional flag to return the oldest messages first.
import CodeSource from "!raw-loader!../../../../../examples/readers/src/simple-directory-reader";
import CodeSource2 from "!raw-loader!../../../../../examples/readers/src/custom-simple-directory-reader";
# Loader
Before you can start indexing your documents, you need to load them into memory.
All "basic" data loaders can be seen below, mapped to their respective filetypes in `SimpleDirectoryReader`. More loaders are shown in the sidebar on the left.
Additionally the following loaders exist without separate documentation:
- `AssemblyAIReader` transcribes audio using [AssemblyAI](https://www.assemblyai.com/).
- [AudioTranscriptReader](../../api/classes/AudioTranscriptReader.md): loads entire transcript as a single document.
- [AudioTranscriptParagraphsReader](../../api/classes/AudioTranscriptParagraphsReader.md): creates a document per paragraph.
- [AudioTranscriptSentencesReader](../../api/classes/AudioTranscriptSentencesReader.md): creates a document per sentence.
- [AudioSubtitlesReader](../../api/classes/AudioTranscriptParagraphsReader.md): creates a document containing the subtitles of a transcript.
- [SimpleMongoReader](../../api/classes/SimpleMongoReader) loads data from a [MongoDB](https://www.mongodb.com/).
Check the [LlamaIndexTS Github](https://github.com/run-llama/LlamaIndexTS) for the most up to date overview of integrations.
## SimpleDirectoryReader
[](https://stackblitz.com/github/run-llama/LlamaIndexTS/tree/main/examples/readers?file=src/simple-directory-reader.ts&title=Simple%20Directory%20Reader)
LlamaIndex.TS supports easy loading of files from folders using the `SimpleDirectoryReader` class.
It is a simple reader that reads all files from a directory and its subdirectories.
<CodeBlock language="ts">{CodeSource}</CodeBlock>
Currently, the following readers are mapped to specific file types:
- `overrideReader` overrides the reader for all file types, including unsupported ones.
- `fileExtToReader` maps a reader to a specific file type. Can override reader for existing file types or add support for new file types.
- `defaultReader` sets a fallback reader for files with unsupported extensions. By default it is `TextFileReader`.
SimpleDirectoryReader supports up to 9 concurrent requests. Use the `numWorkers` option to set the number of concurrent requests. By default it runs in sequential mode, i.e. set to 1.
-`streamingThreshold?`: The threshold for using streaming mode in MB of the JSON Data. CEstimates characters by calculating bytes: `(streamingThreshold * 1024 * 1024) / 2` and comparing against `.length` of the JSON string. Set `undefined` to disable streaming or `0` to always use streaming. Default is `50` MB.
-`ensureAscii?`: Wether to ensure only ASCII characters be present in the output by converting non-ASCII characters to their unicode escape sequence. Default is `false`.
-`isJsonLines?`: Wether the JSON is in JSON Lines format. If true, will split into lines, remove empty one and parse each line as JSON. Note: Uses a custom streaming parser, most likely less robust than json-ext. Default is `false`
-`cleanJson?`: Whether to clean the JSON by filtering out structural characters (`{}, [], and ,`). If set to false, it will just parse the JSON, not removing structural characters. Default is `true`.
-`logger?`: A placeholder for a custom logger function.
Depth-First-Traversal:
-`levelsBack?`: Specifies how many levels up the JSON structure to include in the output. `cleanJson` will be ignored. If set to 0, all levels are included. If undefined, parses the entire JSON, treat each line as an embedding and create a document per top-level array. Default is `undefined`
-`collapseLength?`: The maximum length of JSON string representation to be collapsed into a single line. Only applicable when `levelsBack` is set. Default is `undefined`
LlamaParse `json` mode supports extracting any images found in a page object by using the `getImages` function. They are downloaded to a local folder and can then be sent to a multimodal LLM for further processing.
## Usage
We use the `getImages` method to input our array of JSON objects, download the images to a specified folder and get a list of ImageNodes.
// Split text, create embeddings and query the index
const index = await VectorStoreIndex.fromDocuments(documents);
const queryEngine = index.asQueryEngine();
const response = await queryEngine.query({
query:
"What does the bar graph titled 'Monthly Active Platform Consumers' show?",
});
console.log(response.toString());
}
main().catch(console.error);
```
We use two helper functions to create documents from the text and image nodes provided.
#### Text Documents
To create documents from the text nodes of the json object, we just map the needed values to a new `Document` object. In this case we assign the text as text and the page number as metadata.
```ts
function getTextDocs(jsonList: { text: string; page: number }[]): Document[] {
import CodeSource from "!raw-loader!../../../../../../examples/readers/src/llamaparse";
import CodeSource2 from "!raw-loader!../../../../../../examples/readers/src/simple-directory-reader-with-llamaparse.ts";
# LlamaParse
LlamaParse is an API created by LlamaIndex to efficiently parse files, e.g. it's great at converting PDF tables into markdown.
To use it, first login and get an API key from https://cloud.llamaindex.ai. Make sure to store the key as `apiKey` parameter or in the environment variable `LLAMA_CLOUD_API_KEY`.
Official documentation for LlamaParse can be found [here](https://docs.cloud.llamaindex.ai/).
## Usage
You can then use the `LlamaParseReader` class to load local files and convert them into a parsed document that can be used by LlamaIndex.
See [reader.ts](https://github.com/run-llama/LlamaIndexTS/blob/main/packages/cloud/src/reader.ts) for a list of supported file types:
<CodeBlock language="ts">{CodeSource}</CodeBlock>
### Params
All options can be set with the `LlamaParseReader` constructor.
They can be divided into two groups.
#### General params:
- `apiKey` is required. Can be set as an environment variable `LLAMA_CLOUD_API_KEY`
- `checkInterval` is the interval in seconds to check if the parsing is done. Default is `1`.
- `maxTimeout` is the maximum timeout to wait for parsing to finish. Default is `2000`
- `verbose` shows progress of the parsing. Default is `true`
- `ignoreErrors` set to false to get errors while parsing. Default is `true` and returns an empty array on error.
#### Advanced params:
- `resultType` can be set to `markdown`, `text` or `json`. Defaults to `text`. More information about `json` mode on the next pages.
- `language` primarily helps with OCR recognition. Defaults to `en`. Click [here](../../../api/type-aliases/Language.md) for a list of supported languages.
- `parsingInstructions?` Optional. Can help with complicated document structures. See this [LlamaIndex Blog Post](https://www.llamaindex.ai/blog/launching-the-first-genai-native-document-parsing-platform) for an example.
- `skipDiagonalText?` Optional. Set to true to ignore diagonal text. (Text that is not rotated 0, 90, 180 or 270 degrees)
- `invalidateCache?` Optional. Set to true to ignore the LlamaCloud cache. All document are kept in cache for 48hours after the job was completed to avoid processing the same document twice. Can be useful for testing when trying to re-parse the same document with, e.g. different `parsingInstructions`.
- `doNotCache?` Optional. Set to true to not cache the document.
- `fastMode?` Optional. Set to true to use the fast mode. This mode will skip OCR of images, and table/heading reconstruction. Note: Non-compatible with `gpt4oMode`.
- `doNotUnrollColumns?` Optional. Set to true to keep the text according to document layout. Reduce reconstruction accuracy, and LLMs/embeddings performances in most cases.
- `pageSeparator?` Optional. A templated page separator to use to split the text. If the results contain `{page_number}` (e.g. JSON mode), it will be replaced by the next page number. If not set the default separator `\\n---\\n` will be used.
- `pagePrefix?` Optional. A templated prefix to add to the beginning of each page. If the results contain `{page_number}`, it will be replaced by the page number.
- `pageSuffix?` Optional. A templated suffix to add to the end of each page. If the results contain `{page_number}`, it will be replaced by the page number.
- `gpt4oMode` Deprecated. Use vendorMultimodal params. Set to true to use GPT-4o to extract content. Default is `false`.
- `gpt4oApiKey?` Deprecated. Use vendorMultimodal params. Optional. Set the GPT-4o API key. Lowers the cost of parsing by using your own API key. Your OpenAI account will be charged. Can also be set in the environment variable `LLAMA_CLOUD_GPT4O_API_KEY`.
- `boundingBox?` Optional. Specify an area of the document to parse. Expects the bounding box margins as a string in clockwise order, e.g. `boundingBox = "0.1,0,0,0"` to not parse the top 10% of the document.
- `targetPages?` Optional. Specify which pages to parse by specifying them as a comma-separated list. First page is `0`.
- `splitByPage` Wether to split the results, creating one document per page. Uses the set `pageSeparator` or `\n---\n` as fallback. Default is true.
- `useVendorMultimodalModel` set to true to use a multimodal model. Default is `false`.
- `vendorMultimodalModel?` Optional. Specify which multimodal model to use. Default is GPT4o. See [here](https://docs.cloud.llamaindex.ai/llamaparse/features/multimodal) for a list of available models and cost.
- `vendorMultimodalApiKey?` Optional. Set the multimodal model API key. Can also be set in the environment variable `LLAMA_CLOUD_VENDOR_MULTIMODAL_API_KEY`.
- `numWorkers` as in the python version, is set in `SimpleDirectoryReader`. Default is 1.
### LlamaParse with SimpleDirectoryReader
Below a full example of `LlamaParse` integrated in `SimpleDirectoryReader` with additional options.
// Access the first "pages" (=a single parsed file) object in the array
const jsonList = jsonObjs[0]["pages"];
// Further process the jsonList object as needed.
}
```
### Output
The result format of the response, written to `jsonObjs` in the example, follows this structure:
```json
{
"pages": [
..page objects..
],
"job_metadata": {
"credits_used": int,
"credits_max": int,
"job_credits_usage": int,
"job_pages": int,
"job_is_cache_hit": boolean
},
"job_id": string ,
"file_path": string,
}
}
```
#### Page objects
Within page objects, the following keys may be present depending on your document.
- `page`: The page number of the document.
- `text`: The text extracted from the page.
- `md`: The markdown version of the extracted text.
- `images`: Any images extracted from the page.
- `items`: An array of heading, text and table objects in the order they appear on the page.
### JSON Mode with SimpleDirectoryReader
All Readers share a `loadData` method with `SimpleDirectoryReader` that promises to return a uniform Document with Metadata. This makes JSON mode incompatible with SimpleDirectoryReader.
However, a simple work around is to create a new reader class that extends `LlamaParseReader` and adds a new method or overrides `loadData`, wrapping around JSON mode, extracting the required values, and returning a Document object.
```ts
import { LlamaParseReader, Document } from "llamaindex";
class LlamaParseReaderWithJson extends LlamaParseReader {
// Call loadJson method that was inherited by LlamaParseReader
const jsonObjs = await super.loadJson(filePath);
let documents: Document[] = [];
jsonObjs.forEach((jsonObj) => {
// Making sure it's an array before iterating over it
if (Array.isArray(jsonObj.pages)) {
}
const docs = jsonObj.pages.map(
(page: { text: string; page: number }) =>
new Document({ text: page.text, metadata: { page: page.page } }),
);
documents = documents.concat(docs);
});
return documents;
}
}
```
Now we have documents with page number as metadata. This new reader can be used like any other and be integrated with SimpleDirectoryReader. Since it extends `LlamaParseReader`, you can use the same params.
You can assign any other values of the JSON response to the Document as needed.
Chat stores manage chat history by storing sequences of messages in a structured way, ensuring the order of messages is maintained for accurate conversation flow.
## Available Chat Stores
- [SimpleChatStore](../../../api/classes/SimpleChatStore.md): A simple in-memory chat store with support for [persisting](../index.md#local-storage) data to disk.
Check the [LlamaIndexTS Github](https://github.com/run-llama/LlamaIndexTS) for the most up to date overview of integrations.
Document stores contain ingested document chunks, i.e. [Node](../../documents_and_nodes/index.md)s.
## Available Document Stores
- [SimpleDocumentStore](../../../api/classes/SimpleDocumentStore.md): A simple in-memory document store with support for [persisting](../index.md#local-storage) data to disk.
- [PostgresDocumentStore](../../../api/classes/PostgresDocumentStore.md): A PostgreSQL document store, see [PostgreSQL Storage](../index.md#postgresql-storage).
Check the [LlamaIndexTS Github](https://github.com/run-llama/LlamaIndexTS) for the most up to date overview of integrations.
Index stores are underlying storage components that contain metadata(i.e. information created when indexing) about the [index](../../data_index.md) itself.
## Available Index Stores
- [SimpleIndexStore](../../../api/classes/SimpleIndexStore.md): A simple in-memory index store with support for [persisting](../index.md#local-storage) data to disk.
- [PostgresIndexStore](../../../api/classes/PostgresIndexStore.md): A PostgreSQL index store, , see [PostgreSQL Storage](../index.md#postgresql-storage).
Check the [LlamaIndexTS Github](https://github.com/run-llama/LlamaIndexTS) for the most up to date overview of integrations.
Key-Value Stores represent underlying storage components used in [Document Stores](../doc_stores/index.md) and [Index Stores](../index_stores/index.md)
## Available Key-Value Stores
- [SimpleKVStore](../../../api/classes/SimpleKVStore.md): A simple Key-Value store with support of [persisting](../index.md#local-storage) data to disk.
- [PostgresKVStore](../../../api/classes/PostgresKVStore.md): A PostgreSQL Key-Value store, see [PostgreSQL Storage](../index.md#postgresql-storage).
Check the [LlamaIndexTS Github](https://github.com/run-llama/LlamaIndexTS) for the most up to date overview of integrations.
Vector stores save embedding vectors of your ingested document chunks.
## Available Vector Stores
Available Vector Stores are shown on the sidebar to the left. Additionally the following integrations exist without separate documentation:
- [SimpleVectorStore](../../../api/classes/SimpleVectorStore.md): A simple in-memory vector store with optional [persistance](../index.md#local-storage) to disk.
- [AstraDBVectorStore](../../../api/classes/AstraDBVectorStore.md): A cloud-native, scalable Database-as-a-Service built on Apache Cassandra, see [datastax.com](https://www.datastax.com/products/datastax-astra)
- [ChromaVectorStore](../../../api/classes/ChromaVectorStore.md): An open-source vector database, focused on ease of use and performance, see [trychroma.com](https://www.trychroma.com/)
- [MilvusVectorStore](../../../api/classes/MilvusVectorStore.md): An open-source, high-performance, highly scalable vector database, see [milvus.io](https://milvus.io/)
- [MongoDBAtlasVectorSearch](../../../api/classes/MongoDBAtlasVectorSearch.md): A cloud-based vector search solution for MongoDB, see [mongodb.com](https://www.mongodb.com/products/platform/atlas-vector-search)
- [PGVectorStore](../../../api/classes/PGVectorStore.md): An open-source vector store built on PostgreSQL, see [pgvector Github](https://github.com/pgvector/pgvector)
- [PineconeVectorStore](../../../api/classes/PineconeVectorStore.md): A managed, cloud-native vector database, see [pinecone.io](https://www.pinecone.io/)
- [WeaviateVectorStore](../../../api/classes/WeaviateVectorStore.md): An open-source, ai-native vector database, see [weaviate.io](https://weaviate.io/)
Check the [LlamaIndexTS Github](https://github.com/run-llama/LlamaIndexTS) for the most up to date overview of integrations.
`Document`s and `Node`s are the basic building blocks of any index. While the API for these objects is similar, `Document` objects represent entire files, while `Node`s are smaller pieces of that original document, that are suitable for an LLM and Q&A.
By default, DeepInfraEmbedding is using the sentence-transformers/clip-ViT-B-32 model. You can change the model by passing the model parameter to the constructor.
For example:
```ts
import{DeepInfraEmbedding}from"llamaindex";
constmodel="intfloat/e5-large-v2";
Settings.embedModel=newDeepInfraEmbedding({
model,
});
```
You can also set the `maxRetries` and `timeout` parameters when initializing `DeepInfraEmbedding` for better control over the request behavior.
Per default, `HuggingFaceEmbedding` is using the `Xenova/all-MiniLM-L6-v2` model. You can change the model by passing the `modelType` parameter to the constructor.
If you're not using a quantized model, set the `quantized` parameter to `false`.
For example, to use the not quantized `BAAI/bge-small-en-v1.5` model, you can use the following code:
Welcome to the mixedbread embeddings guide! This guide will help you use the mixedbread ai's API to generate embeddings for your text documents, ensuring you get the most relevant information, just like picking the freshest bread from the bakery.
To find out more about the latest features, updates, and available models, visit [mixedbread.ai](https://mixedbread-ai.com/).
## Table of Contents
1. [Setup](#setup)
2. [Usage with LlamaIndex](#usage-with-llamaindex)
3. [Embeddings with Custom Parameters](#embeddings-with-custom-parameters)
## Setup
First, you will need to install the `llamaindex` package.
```bash
pnpm install llamaindex
```
Next, sign up for an API key at [mixedbread.ai](https://mixedbread.ai/). Once you have your API key, you can import the necessary modules and create a new instance of the `MixedbreadAIEmbeddings` class.
This section will guide you through integrating mixedbread embeddings with LlamaIndex for more advanced usage.
### Step 1: Load and Index Documents
For this example, we will use a single document. In a real-world scenario, you would have multiple documents to index, like a variety of breads in a bakery.
Combine the retriever and the embed model to create a query engine. This setup ensures that your queries are processed to provide the best results, like arranging the bread in the order of freshness and quality.
Models can require prompts to generate embeddings for queries, in the 'mixedbread-ai/mxbai-embed-large-v1' model's case, the prompt is `Represent this sentence for searching relevant passages:`.
```ts
constqueryEngine=index.asQueryEngine();
constquery=
"Represent this sentence for searching relevant passages: What is bread?";
// Log the response
constresults=awaitqueryEngine.query(query);
console.log(results);// Serving up the freshest, most relevant results.
```
## Embeddings with Custom Parameters
This section will guide you through generating embeddings with custom parameters and usage with f.e. matryoshka and binary embeddings.
### Step 1: Create an Instance of MixedbreadAIEmbeddings
Create a new instance of the `MixedbreadAIEmbeddings` class with custom parameters. For example, to use the `mixedbread-ai/mxbai-embed-large-v1` model with a batch size of 64, normalized embeddings, and binary encoding format:
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
github-actions[bot]
