`@xenova/transformers` only ship node.js and browser output, it's not possible to load this in edge runtime and workerd
This reverts commit 34fb1d8992.
* refactor: merge HistoryChatEngine and ContextChatEngine and use ChatHistory for all chat engines
* fix: add safeguard for tokensToSummarize
* refactor: unfold chat engines to own folder
* refactor: extract LLM types
* refactor: move multi-modal types to llm
* docs(changeset): Remove HistoryChatEngine and use ChatHistory for all chat engines
* dev: add debug launcher and don't lint generated code
- expanded LLM interface
- cleaned up OpenAI LLM stream
- simplified types in CallbackManager
-> CallbackResponse should require Token I think, we already have the StreamToken inside of the LLM's stream_chat anyways
- SubQuestionOutputParser is not working as expected, writing some tests to check it out.
QuestionGenerator outputs a list of objects in string format, which is unexpected.
In particular:
"[{prompt: "smth", response: "nothing"}]"
Allows our scripts to be run repeatedly with vector store backing and
persistence without throwing an error.
This is rudimentary support which doesn't work if the document has
changed over time.
Also added the insert and delete functions so that documents can be
added manually.
In theory this will stop the API docs thrashing on every build.
Of course, there is a drawback in that if main changes but we haven't
released a new version of the docs yet the links will go out of date.
So longer term we might want to investigate some kind of variable where
we can continue to have up to date revs but keep the rev in a single
variable somewhere:
https://github.com/facebook/docusaurus/issues/395
In the meantime much smaller commits will be a relief
description:Write something like "Modify the ... api endpoint to use ... version and ... framework" If you would like to use sweep.dev prefix with "Sweep:"
body:
- type:textarea
id:description
attributes:
label:Details
description:More details
placeholder:We are migrating this function to ... version because ...
packages/llamaindex which is the main NPM library llamaindex
examples is where the demo code lives
### Turborepo docs
You can checkout how Turborepo works using the default [README-turborepo.md](/README-turborepo.md)
## Getting Started
Install NodeJS. Preferably v18 using nvm or n.
Inside the LlamaIndexTS directory:
```
npm i -g pnpm ts-node
pnpm install
```
Note: we use pnpm in this repo, which has a lot of the same functionality and CLI options as npm but it does do some things better in a monorepo, like centralizing dependencies and caching.
PNPM's has documentation on its [workspace feature](https://pnpm.io/workspaces) and Turborepo had some [useful documentation also](https://turbo.build/repo/docs/core-concepts/monorepos/running-tasks).
### Running Typescript
When we publish to NPM we will have a tsc compiled version of the library in JS. For now, the easiest thing to do is use ts-node.
### Test cases
To run them, run
```
pnpm run test
```
To write new test cases write them in [packages/llamaindex/tests](/packages/llamaindex/tests)
We use Jest https://jestjs.io/ to write our test cases. Jest comes with a bunch of built in assertions using the expect function: https://jestjs.io/docs/expect
### Demo applications
There is an existing ["example"](/examples/README.md) demos folder with mainly NodeJS scripts. Feel free to add additional demos to that folder. If you would like to try out your changes in the core package with a new demo, you need to run the build command in the README.
You can create new demo applications in the apps folder. Just run pnpm init in the folder after you create it to create its own package.json
### Installing packages
To install packages for a specific package or demo application, run
```
pnpm add [NPM Package] --filter [package or application i.e. llamaindex or docs]
```
To install packages for every package or application run
```
pnpm add -w [NPM Package]
```
### Docs
To contribute to the docs, go to the docs website folder and run the Docusaurus instance.
```bash
cd apps/docs
pnpm install
pnpm start
```
That should start a webserver which will serve the docs on https://localhost:3000
Any changes you make should be reflected in the browser. If you need to regenerate the API docs and find that your TSDoc isn't getting the updates, feel free to remove apps/docs/api. It will automatically regenerate itself when you run pnpm start again.
## Changeset
We use [changesets](https://github.com/changesets/changesets) for managing versions and changelogs. To create a new changeset, run:
```
pnpm changeset
```
Please send a descriptive changeset for each PR.
## Publishing (maintainers only)
The [Release Github Action](.github/workflows/release.yml) is automatically generating and updating a
PR called "Release {version}".
This PR will update the `package.json` and `CHANGELOG.md` files of each package according to
the current changesets in the [.changeset](.changeset/) folder.
If this PR is merged it will automatically add version tags to the repository and publish the updated packages to NPM.
This Turborepo includes the following packages/apps:
### Apps and Packages
-`docs`: a [Next.js](https://nextjs.org/) app
-`web`: another [Next.js](https://nextjs.org/) app
-`ui`: a stub React component library shared by both `web` and `docs` applications
-`eslint-config-custom`: `eslint` configurations (includes `eslint-config-next` and `eslint-config-prettier`)
-`tsconfig`: `tsconfig.json`s used throughout the monorepo
Each package/app is 100% [TypeScript](https://www.typescriptlang.org/).
### Utilities
This Turborepo has some additional tools already setup for you:
- [TypeScript](https://www.typescriptlang.org/) for static type checking
- [ESLint](https://eslint.org/) for code linting
- [Prettier](https://prettier.io) for code formatting
### Build
To build all apps and packages, run the following command:
```
cd my-turborepo
pnpm build
```
### Develop
To develop all apps and packages, run the following command:
```
cd my-turborepo
pnpm dev
```
### Remote Caching
Turborepo can use a technique known as [Remote Caching](https://turbo.build/repo/docs/core-concepts/remote-caching) to share cache artifacts across machines, enabling you to share build caches with your team and CI/CD pipelines.
By default, Turborepo will cache locally. To enable Remote Caching you will need an account with Vercel. If you don't have an account you can [create one](https://vercel.com/signup), then enter the following commands:
```
cd my-turborepo
npx turbo login
```
This will authenticate the Turborepo CLI with your [Vercel account](https://vercel.com/docs/concepts/personal-accounts/overview).
Next, you can link your Turborepo to your Remote Cache by running the following command from the root of your Turborepo:
LlamaIndex is a data framework for your LLM application.
Right now there are two packages of importance:
Use your own data with large language models (LLMs, OpenAI ChatGPT and others) in Typescript and Javascript.
packages/core which is the main NPM library @llamaindex/core
Documentation: https://ts.llamaindex.ai/
apps/simple is where the demo code lives
Try examples online:
### Turborepo docs
[](https://stackblitz.com/github/run-llama/LlamaIndexTS/tree/main/examples)
You can checkout how Turborepo works using the built in [README-turborepo.md](README-turborepo.md)
## What is LlamaIndex.TS?
## Getting Started
LlamaIndex.TS aims to be a lightweight, easy to use set of libraries to help you integrate large language models into your applications with your own data.
For now, browser support is limited due to the lack of support for [AsyncLocalStorage-like APIs](https://github.com/tc39/proposal-async-context)
## Getting started
```shell
npm install llamaindex
pnpm install llamaindex
yarn add llamaindex
jsr install @llamaindex/core
```
Note: we use pnpm in this repo, which has a lot of the same functionality and CLI options as npm but it does do some things better in a monorepo, like centralizing dependencies and caching.
### Node.js
PNPM's has documentation on its [workspace feature](https://pnpm.io/workspaces) and Turborepo had some [useful documentation also](https://turbo.build/repo/docs/core-concepts/monorepos/running-tasks).
To write new test cases write them in packages/core/src/tests
We use Jest https://jestjs.io/ to write our test cases. Jest comes with a bunch of built in assertions using the expect function: https://jestjs.io/docs/expect
### Demo applications
You can create new demo applications in the apps folder. Just run pnpm init in the folder after you create it to create its own package.json
### Installing packages
To install packages for a specific package or demo application, run
```
pnpm add [NPM Package] --filter [package or application i.e. core or simple]
```bash
# `pnpm install tsx` before running the script
node --import tsx ./main.ts
```
To install packages for every package or application run
### Next.js
First, you will need to add a llamaindex plugin to your Next.js project.
```js
// next.config.js
constwithLlamaIndex=require("llamaindex/next");
module.exports=withLlamaIndex({
// your next.js config
});
```
pnpm add -w [NPM Package]
You can combine `ai` with `llamaindex` in Next.js with RSC (React Server Components).
```tsx
// src/apps/page.tsx
"use client";
import{chatWithAgent}from"@/actions";
importtype{JSX}from"react";
import{useFormState}from"react-dom";
// You can use the Edge runtime in Next.js by adding this line:
Check out our NextJS playground at https://llama-playground.vercel.app/. The source is available at https://github.com/run-llama/ts-playground
## Core concepts for getting started:
- [Document](/packages/llamaindex/src/Node.ts): A document represents a text file, PDF file or other contiguous piece of data.
- [Node](/packages/llamaindex/src/Node.ts): The basic data building block. Most commonly, these are parts of the document split into manageable pieces that are small enough to be fed into an embedding model and LLM.
- [Embedding](/packages/llamaindex/src/embeddings/OpenAIEmbedding.ts): Embeddings are sets of floating point numbers which represent the data in a Node. By comparing the similarity of embeddings, we can derive an understanding of the similarity of two pieces of data. One use case is to compare the embedding of a question with the embeddings of our Nodes to see which Nodes may contain the data needed to answer that quesiton. Because the default service context is OpenAI, the default embedding is `OpenAIEmbedding`. If using different models, say through Ollama, use this [Embedding](/packages/llamaindex/src/embeddings/OllamaEmbedding.ts) (see all [here](/packages/llamaindex/src/embeddings)).
- [Indices](/packages/llamaindex/src/indices/): Indices store the Nodes and the embeddings of those nodes. QueryEngines retrieve Nodes from these Indices using embedding similarity.
- [QueryEngine](/packages/llamaindex/src/engines/query/RetrieverQueryEngine.ts): Query engines are what generate the query you put in and give you back the result. Query engines generally combine a pre-built prompt with selected Nodes from your Index to give the LLM the context it needs to answer your query. To build a query engine from your Index (recommended), use the [`asQueryEngine`](/packages/llamaindex/src/indices/BaseIndex.ts) method on your Index. See all query engines [here](/packages/llamaindex/src/engines/query).
- [ChatEngine](/packages/llamaindex/src/engines/chat/SimpleChatEngine.ts): A ChatEngine helps you build a chatbot that will interact with your Indices. See all chat engines [here](/packages/llamaindex/src/engines/chat).
- [SimplePrompt](/packages/llamaindex/src/Prompt.ts): A simple standardized function call definition that takes in inputs and formats them in a template literal. SimplePrompts can be specialized using currying and combined using other SimplePrompt functions.
## Tips when using in non-Node.js environments
When you are importing `llamaindex` in a non-Node.js environment(such as React Server Components, Cloudflare Workers, etc.)
Some classes are not exported from top-level entry file.
The reason is that some classes are only compatible with Node.js runtime,(e.g. `PDFReader`) which uses Node.js specific APIs(like `fs`, `child_process`, `crypto`).
If you need any of those classes, you have to import them instead directly though their file path in the package.
Here's an example for importing the `PineconeVectorStore` class:
Open [http://localhost:3000](http://localhost:3000) with your browser to see the result.
You can start editing the page by modifying `pages/index.js`. The page auto-updates as you edit the file.
[API routes](https://nextjs.org/docs/api-routes/introduction) can be accessed on [http://localhost:3000/api/hello](http://localhost:3000/api/hello). This endpoint can be edited in `pages/api/hello.js`.
The `pages/api` directory is mapped to `/api/*`. Files in this directory are treated as [API routes](https://nextjs.org/docs/api-routes/introduction) instead of React pages.
## Learn More
To learn more about Next.js, take a look at the following resources:
- [Next.js Documentation](https://nextjs.org/docs) - learn about Next.js features and API.
- [Learn Next.js](https://nextjs.org/learn/foundations/about-nextjs) - an interactive Next.js tutorial.
You can check out [the Next.js GitHub repository](https://github.com/vercel/next.js/) - your feedback and contributions are welcome!
## Deploy on Vercel
The easiest way to deploy your Next.js app is to use the [Vercel Platform](https://vercel.com/new?utm_source=github.com&utm_medium=referral&utm_campaign=turborepo-readme) from the creators of Next.js.
Check out our [Next.js deployment documentation](https://nextjs.org/docs/deployment) for more details.
import CodeSource from "!raw-loader!../../../../examples/chatEngine";
# Chat Engine
Chat Engine is a class that allows you to create a chatbot from a retriever. It is a wrapper around a retriever that allows you to chat with it in a conversational manner.
LlamaIndex.TS supports OpenAI and [other remote LLM APIs](other_llms). You can also run a local LLM on your machine!
## Using a local model via Ollama
The easiest way to run a local LLM is via the great work of our friends at [Ollama](https://ollama.com/), who provide a simple to use client that will download, install and run a [growing range of models](https://ollama.com/library) for you.
### Install Ollama
They provide a one-click installer for Mac, Linux and Windows on their [home page](https://ollama.com/).
### Pick and run a model
Since we're going to be doing agentic work, we'll need a very capable model, but the largest models are hard to run on a laptop. We think `mixtral 8x7b` is a good balance between power and resources, but `llama3` is another great option. You can run Mixtral by running
```bash
ollama run mixtral:8x7b
```
The first time you run it will also automatically download and install the model for you.
### Switch the LLM in your code
To tell LlamaIndex to use a local LLM, use the `Settings` object:
```javascript
Settings.llm = new Ollama({
model: "mixtral:8x7b",
});
```
### Use local embeddings
If you're doing retrieval-augmented generation, LlamaIndex.TS will also call out to OpenAI to index and embed your data. To be entirely local, you can use a local embedding model like this:
```javascript
Settings.embedModel = new HuggingFaceEmbedding({
modelType: "BAAI/bge-small-en-v1.5",
quantized: false,
});
```
The first time this runs it will download the embedding model to run it.
### Try it out
With a local LLM and local embeddings in place, you can perform RAG as usual and everything will happen on your machine without calling an API:
Our GitHub repository has a wealth of examples to explore and try out. You can check out our [examples folder](https://github.com/run-llama/LlamaIndexTS/tree/main/examples) to see them all at once, or browse the pages in this section for some selected highlights.
## Check out all examples
It may be useful to check out all the examples at once so you can try them out locally. To do this into a folder called `my-new-project`, run these commands:
import CodeSource from "!raw-loader!../../../../examples/mistral";
# Using other LLM APIs
By default LlamaIndex.TS uses OpenAI's LLMs and embedding models, but we support [lots of other LLMs](../modules/llms) including models from Mistral (Mistral, Mixtral), Anthropic (Claude) and Google (Gemini).
If you don't want to use an API at all you can [run a local model](../../examples/local_llm)
## Using another LLM
You can specify what LLM LlamaIndex.TS will use on the `Settings` object, like this:
```typescript
import { MistralAI, Settings } from "llamaindex";
Settings.llm = new MistralAI({
model: "mistral-tiny",
apiKey: "<YOUR_API_KEY>",
});
```
You can see examples of other APIs we support by checking out "Available LLMs" in the sidebar of our [LLMs section](../modules/llms).
## Using another embedding model
A frequent gotcha when trying to use a different API as your LLM is that LlamaIndex will also by default index and embed your data using OpenAI's embeddings. To completely switch away from OpenAI you will need to set your embedding model as well, for example:
```typescript
import { MistralAIEmbedding, Settings } from "llamaindex";
Settings.embedModel = new MistralAIEmbedding();
```
We support [many different embeddings](../modules/embeddings).
## Full example
This example uses Mistral's `mistral-tiny` model as the LLM and Mistral for embeddings as well.
A data connector (i.e. `Reader`) ingest data from different data sources and data formats into a simple `Document` representation (text and simple metadata).
[**Documents / Nodes**](../modules/documents_and_nodes/index.md): A `Document` is a generic container around any data source - for instance, a PDF, an API output, or retrieved data from a database. A `Node` is the atomic unit of data in LlamaIndex and represents a "chunk" of a source `Document`. It's a rich representation that includes metadata and relationships (to other nodes) to enable accurate and expressive retrieval operations.
[**Data Indexes**](../modules/data_index.md):
Once you've ingested your data, LlamaIndex helps you index data into a format that's easy to retrieve.
Under the hood, LlamaIndex parses the raw documents into intermediate representations, calculates vector embeddings, and stores your data in-memory or to disk.
### Querying Stage
In the querying stage, the query pipeline retrieves the most relevant context given a user query,
and pass that to the LLM (along with the query) to synthesize a response.
This gives the LLM up-to-date knowledge that is not in its original training data,
(also reducing hallucination).
The key challenge in the querying stage is retrieval, orchestration, and reasoning over (potentially many) knowledge bases.
LlamaIndex provides composable modules that help you build and integrate RAG pipelines for Q&A (query engine), chatbot (chat engine), or as part of an agent.
These building blocks can be customized to reflect ranking preferences, as well as composed to reason over multiple knowledge bases in a structured way.
#### Building Blocks
[**Retrievers**](../modules/retriever.md):
A retriever defines how to efficiently retrieve relevant context from a knowledge base (i.e. index) when given a query.
The specific retrieval logic differs for difference indices, the most popular being dense retrieval against a vector index.
We support Node.JS versions 18, 20 and 22, with experimental support for Deno, Bun and Vercel Edge functions.
## Installation from NPM
```bash npm2yarn
npm install llamaindex
```
### Environment variables
Our examples use OpenAI by default. You can use [other LLMs](../examples/other_llms) via their APIs; if you would prefer to use local models check out our [local LLM example](../examples/local_llm).
To use OpenAI, you'll need to [get an OpenAI API key](https://platform.openai.com/account/api-keys) and then make it available as an environment variable this way:
```bash
export OPENAI_API_KEY="sk-......" # Replace with your key
```
If you want to have it automatically loaded every time, add it to your `.zshrc/.bashrc`.
**WARNING:** do not check in your OpenAI key into version control. GitHub automatically invalidates OpenAI keys checked in by accident.
## What next?
- The easiest way to started is to [build a full-stack chat app with `create-llama`](starter_tutorial/chatbot).
- Try our other [getting started tutorials](starter_tutorial/retrieval_augmented_generation)
- Learn more about the [high level concepts](concepts) behind how LlamaIndex works
- Check out our [many examples](../examples/more_examples) of LlamaIndex.TS in action
import CodeSource from "!raw-loader!../../../../../examples/agent/openai";
# Agent tutorial
We have a comprehensive, step-by-step [guide to building agents in LlamaIndex.TS](../../guides/agents/setup) that we recommend to learn what agents are and how to build them for production. But building a basic agent is simple:
## Set up
In a new folder:
```bash npm2yarn
npm init
npm install -D typescript @types/node
```
## Run agent
Create the file `example.ts`. This code will:
- Create two tools for use by the agent:
- A `sumNumbers` tool that adds two numbers
- A `divideNumbers` tool that divides numbers
-
- Give an example of the data structure we wish to generate
- Prompt the LLM with instructions and the example, plus a sample transcript
<CodeBlock language="ts">{CodeSource}</CodeBlock>
To run the code:
```bash
npx tsx example.ts
```
You should expect output something like:
```
{
content: 'The sum of 5 + 5 is 10. When you divide 10 by 2, you get 5.',
Once you've mastered basic [retrieval-augment generation](retrieval_augmented_generation) you may want to create an interface to chat with your data. You can do this step-by-step, but we recommend getting started quickly using `create-llama`.
## Using create-llama
`create-llama` is a powerful but easy to use command-line tool that generates a working, full-stack web application that allows you to chat with your data. You can learn more about it on [the `create-llama` README page](https://www.npmjs.com/package/create-llama).
Run it once and it will ask you a series of questions about the kind of application you want to generate. Then you can customize your application to suit your use-case. To get started, run:
```bash npm2yarn
npx create-llama@latest
```
Once your app is generated, `cd` into your app directory and run
```bash npm2yarn
npm run dev
```
to start the development server. You can then visit [http://localhost:3000](http://localhost:3000) to see your app, which should look something like this:
import CodeSource from "!raw-loader!../../../../../examples/vectorIndex";
import TSConfigSource from "!!raw-loader!../../../../../examples/tsconfig.json";
# Retrieval Augmented Generation (RAG) Tutorial
One of the most common use-cases for LlamaIndex is Retrieval-Augmented Generation or RAG, in which your data is indexed and selectively retrieved to be given to an LLM as source material for responding to a query. You can learn more about the [concepts behind RAG](../concepts).
## Set up the project
In a new folder, run:
```bash npm2yarn
npm init
npm install -D typescript @types/node
```
Then, check out the [installation](../installation) steps to install LlamaIndex.TS and prepare an OpenAI key.
You can use [other LLMs](../../examples/other_llms) via their APIs; if you would prefer to use local models check out our [local LLM example](../../examples/local_llm).
## Run queries
Create the file `example.ts`. This code will
- load an example file
- convert it into a Document object
- index it (which creates embeddings using OpenAI)
- create a query engine to answer questions about the data
In college, the author studied subjects like linear algebra and physics, but did not find them particularly interesting. They started slacking off, skipping lectures, and eventually stopped attending classes altogether. They also had a negative experience with their English classes, where they were required to pay for catch-up training despite getting verbal approval to skip most of the classes. Ultimately, the author lost motivation for college due to their job as a software developer and stopped attending classes, only returning years later to pick up their papers.
0: Score: 0.8305309270895813 - I started this decade as a first-year college stud...
1: Score: 0.8286388215713089 - A short digression. I’m not saying colleges are wo...
```
Once you've mastered basic RAG, you may want to consider [chatting with your data](chatbot).
import CodeSource from "!raw-loader!../../../../../examples/jsonExtract";
# Structured data extraction tutorial
Make sure you have installed LlamaIndex.TS and have an OpenAI key. If you haven't, check out the [installation](../installation) guide.
You can use [other LLMs](../../examples/other_llms) via their APIs; if you would prefer to use local models check out our [local LLM example](../../examples/local_llm).
## Set up
In a new folder:
```bash npm2yarn
npm init
npm install -D typescript @types/node
```
## Extract data
Create the file `example.ts`. This code will:
- Set up an LLM connection to GPT-4
- Give an example of the data structure we wish to generate
- Prompt the LLM with instructions and the example, plus a sample transcript
<CodeBlock language="ts">{CodeSource}</CodeBlock>
To run the code:
```bash
npx tsx example.ts
```
You should expect output something like:
```json
{
"summary": "Sarah from XYZ Company called John to introduce the XYZ Widget, a tool designed to automate tasks and improve productivity. John expressed interest and requested case studies and a product demo. Sarah agreed to send the information and follow up to schedule the demo.",
"products": ["XYZ Widget"],
"rep_name": "Sarah",
"prospect_name": "John",
"action_items": [
"Send case studies and additional product information to John",
In this guide we'll walk you through the process of building an Agent in JavaScript using the LlamaIndex.TS library, starting from nothing and adding complexity in stages.
## What is an Agent?
In LlamaIndex, an agent is a semi-autonomous piece of software powered by an LLM that is given a task and executes a series of steps towards solving that task. It is given a set of tools, which can be anything from arbitrary functions up to full LlamaIndex query engines, and it selects the best available tool to complete each step. When each step is completed, the agent judges whether the task is now complete, in which case it returns a result to the user, or whether it needs to take another step, in which case it loops back to the start.
## Install LlamaIndex.TS
You'll need to have a recent version of [Node.js](https://nodejs.org/en) installed. Then you can install LlamaIndex.TS by running
```bash
npm install llamaindex
```
## Choose your model
By default we'll be using OpenAI with GPT-4, as it's a powerful model and easy to get started with. If you'd prefer to run a local model, see [using a local model](local_model).
## Get an OpenAI API key
If you don't already have one, you can sign up for an [OpenAI API key](https://platform.openai.com/api-keys). You should then put the key in a `.env` file in the root of the project; the file should look like
```
OPENAI_API_KEY=sk-XXXXXXXXXXXXXXXXXXXXXXXX
```
We'll use `dotenv` to pull the API key out of that .env file, so also run:
```bash
npm install dotenv
```
Now you're ready to [create your agent](create_agent).
We want to use `await` so we're going to wrap all of our code in a `main` function, like this:
```typescript
// Your imports go here
async function main() {
// the rest of your code goes here
}
main().catch(console.error);
```
For the rest of this guide we'll assume your code is wrapped like this so we can use `await`. You can run the code this way:
```bash
npx tsx example.ts
```
### Load your dependencies
First we'll need to pull in our dependencies. These are:
- The OpenAI class to use the OpenAI LLM
- FunctionTool to provide tools to our agent
- OpenAIAgent to create the agent itself
- Settings to define some global settings for the library
- Dotenv to load our API key from the .env file
```javascript
import { OpenAI, FunctionTool, OpenAIAgent, Settings } from "llamaindex";
import "dotenv/config";
```
### Initialize your LLM
We need to tell our OpenAI class where its API key is, and which of OpenAI's models to use. We'll be using `gpt-4o`, which is capable while still being pretty cheap. This is a global setting, so anywhere an LLM is needed will use the same model.
```javascript
Settings.llm = new OpenAI({
apiKey: process.env.OPENAI_API_KEY,
model: "gpt-4o",
});
```
### Turn on logging
We want to see what our agent is up to, so we're going to hook into some events that the library generates and print them out. There are several events possible, but we'll specifically tune in to `llm-tool-call` (when a tool is called) and `llm-tool-result` (when it responds).
We're going to create a very simple function that adds two numbers together. This will be the tool we ask our agent to use.
```javascript
const sumNumbers = ({ a, b }) => {
return `${a + b}`;
};
```
Note that we're passing in an object with two named parameters, `a` and `b`. This is a little unusual, but important for defining a tool that an LLM can use.
### Turn the function into a tool for the agent
This is the most complicated part of creating an agent. We need to define a `FunctionTool`. We have to pass in:
- The function itself (`sumNumbers`)
- A name for the function, which the LLM will use to call it
- A description of the function. The LLM will read this description to figure out what the tool does, and if it needs to call it
- A schema for function. We tell the LLM that the parameter is an `object`, and we tell it about the two named parameters we gave it, `a` and `b`. We describe each parameter as a `number`, and we say that both are required.
- You can see [more examples of function schemas](https://cookbook.openai.com/examples/how_to_call_functions_with_chat_models).
```javascript
const tool = FunctionTool.from(sumNumbers, {
name: "sumNumbers",
description: "Use this function to sum two numbers",
parameters: {
type: "object",
properties: {
a: {
type: "number",
description: "First number to sum",
},
b: {
type: "number",
description: "Second number to sum",
},
},
required: ["a", "b"],
},
});
```
We then wrap up the tools into an array. We could provide lots of tools this way, but for this example we're just using the one.
```javascript
const tools = [tool];
```
### Create the agent
With your LLM already set up and your tools defined, creating an agent is simple:
```javascript
const agent = new OpenAIAgent({ tools });
```
### Ask the agent a question
We can use the `chat` interface to ask our agent a question, and it will use the tools we've defined to find an answer.
```javascript
let response = await agent.chat({
message: "Add 101 and 303",
});
console.log(response);
```
Let's see what running this looks like using `npx tsx agent.ts`
We're seeing two pieces of output here. The first is our callback firing when the tool is called. You can see in `toolResult` that the LLM has correctly passed `101` and `303` to our `sumNumbers` function, which adds them up and returns `404`.
The second piece of output is the response from the LLM itself, where the `message.content` key is giving us the answer.
Great! We've built an agent with tool use! Next you can:
- [See the full code](https://github.com/run-llama/ts-agents/blob/main/1_agent/agent.ts)
- [Switch to a local LLM](local_model)
- Move on to [add Retrieval-Augmented Generation to your agent](agentic_rag)
If you're happy using OpenAI, you can skip this section, but many people are interested in using models they run themselves. The easiest way to do this is via the great work of our friends at [Ollama](https://ollama.com/), who provide a simple to use client that will download, install and run a [growing range of models](https://ollama.com/library) for you.
### Install Ollama
They provide a one-click installer for Mac, Linux and Windows on their [home page](https://ollama.com/).
### Pick and run a model
Since we're going to be doing agentic work, we'll need a very capable model, but the largest models are hard to run on a laptop. We think `mixtral 8x7b` is a good balance between power and resources, but `llama3` is another great option. You can run it simply by running
```bash
ollama run mixtral:8x7b
```
The first time you run it will also automatically download and install the model for you.
### Switch the LLM in your code
There are two changes you need to make to the code we already wrote in `1_agent` to get Mixtral 8x7b to work. First, you need to switch to that model. Replace the call to `Settings.llm` with this:
```javascript
Settings.llm = new Ollama({
model: "mixtral:8x7b",
});
```
### Swap to a ReActAgent
In our original code we used a specific OpenAIAgent, so we'll need to switch to a more generic agent pattern, the ReAct pattern. This is simple: change the `const agent` line in your code to read
```javascript
const agent = new ReActAgent({ tools });
```
(You will also need to bring in `Ollama` and `ReActAgent` in your imports)
### Run your totally local agent
Because your embeddings were already local, your agent can now run entirely locally without making any API calls.
```bash
node agent.mjs
```
Note that your model will probably run a lot slower than OpenAI, so be prepared to wait a while!
**_Output_**
```javascript
{
response: {
message: {
role: 'assistant',
content: ' Thought: I need to use a tool to add the numbers 101 and 303.\n' +
'Action: sumNumbers\n' +
'Action Input: {"a": 101, "b": 303}\n' +
'\n' +
'Observation: 404\n' +
'\n' +
'Thought: I can answer without using any more tools.\n' +
'Answer: The sum of 101 and 303 is 404.'
},
raw: {
model: 'mixtral:8x7b',
created_at: '2024-05-09T00:24:30.339473Z',
message: [Object],
done: true,
total_duration: 64678371209,
load_duration: 57394551334,
prompt_eval_count: 475,
prompt_eval_duration: 4163981000,
eval_count: 94,
eval_duration: 3116692000
}
},
sources: [Getter]
}
```
Tada! You can see all of this in the folder `1a_mixtral`.
### Extending to other examples
You can use a ReActAgent instead of an OpenAIAgent in any of the further examples below, but keep in mind that GPT-4 is a lot more capable than Mixtral 8x7b, so you may see more errors or failures in reasoning if you are using an entirely local setup.
### Next steps
Now you've got a local agent, you can [add Retrieval-Augmented Generation to your agent](agentic_rag).
While an agent that can perform math is nifty (LLMs are usually not very good at math), LLM-based applications are always more interesting when they work with large amounts of data. In this case, we're going to use a 200-page PDF of the proposed budget of the city of San Francisco for fiscal years 2024-2024 and 2024-2025. It's a great example because it's extremely wordy and full of tables of figures, which present a challenge for humans and LLMs alike.
To learn more about RAG, we recommend this [introduction](https://docs.llamaindex.ai/en/stable/getting_started/concepts/) from our Python docs. We'll assume you know the basics:
- You need to parse your source data into chunks of text
- You need to encode that text as numbers, called embeddings
- You need to search your embeddings for the most relevant chunks of text
- You feed your relevant chunks and a query to an LLM to answer a question
We're going to start with the same agent we [built in step 1](https://github.com/run-llama/ts-agents/blob/main/1_agent/agent.ts), but make a few changes. You can find the finished version [in the repository](https://github.com/run-llama/ts-agents/blob/main/2_agentic_rag/agent.ts).
### New dependencies
We'll be bringing in `SimpleDirectoryReader`, `HuggingFaceEmbedding`, `VectorStoreIndex`, and `QueryEngineTool` from LlamaIndex.TS, as well as the dependencies we previously used.
```javascript
import {
OpenAI,
FunctionTool,
OpenAIAgent,
Settings,
SimpleDirectoryReader,
HuggingFaceEmbedding,
VectorStoreIndex,
QueryEngineTool,
} from "llamaindex";
```
### Add an embedding model
To encode our text into embeddings, we'll need an embedding model. We could use OpenAI for this but to save on API calls we're going to use a local embedding model from HuggingFace.
```javascript
Settings.embedModel = new HuggingFaceEmbedding({
modelType: "BAAI/bge-small-en-v1.5",
quantized: false,
});
```
### Load data using SimpleDirectoryReader
SimpleDirectoryReader is a flexible tool that can read a variety of file formats. We're going to point it at our data directory, which contains just the single PDF file, and get it to return a set of documents.
Now we turn our text into embeddings. The `VectorStoreIndex` class takes care of this for us when we use the `fromDocuments` method (it uses the embedding model we defined in `Settings` earlier).
```javascript
const index = await VectorStoreIndex.fromDocuments(documents);
```
### Configure a retriever
Before LlamaIndex can send a query to the LLM, it needs to find the most relevant chunks to send. That's the purpose of a `Retriever`. We're going to get `VectorStoreIndex` to act as a retriever for us
```javascript
const retriever = await index.asRetriever();
```
### Configure how many documents to retrieve
By default LlamaIndex will retrieve just the 2 most relevant chunks of text. This document is complex though, so we'll ask for more context.
```javascript
retriever.similarityTopK = 10;
```
### Create a query engine
And our final step in creating a RAG pipeline is to create a query engine that will use the retriever to find the most relevant chunks of text, and then use the LLM to answer the question.
```javascript
const queryEngine = await index.asQueryEngine({
retriever,
});
```
### Define the query engine as a tool
Just as before we created a `FunctionTool`, we're going to create a `QueryEngineTool` that uses our `queryEngine`.
```javascript
const tools = [
new QueryEngineTool({
queryEngine: queryEngine,
metadata: {
name: "san_francisco_budget_tool",
description: `This tool can answer detailed questions about the individual components of the budget of San Francisco in 2023-2024.`,
},
}),
];
```
As before, we've created an array of tools with just one tool in it. The metadata is slightly different: we don't need to define our parameters, we just give the tool a name and a natural-language description.
### Create the agent as before
Creating the agent and asking a question is exactly the same as before, but we'll ask a different question.
```javascript
// create the agent
const agent = new OpenAIAgent({ tools });
let response = await agent.chat({
message: "What's the budget of San Francisco in 2023-2024?",
});
console.log(response);
```
Once again we'll run `npx tsx agent.ts` and see what we get:
**_Output_**
```javascript
{
toolCall: {
id: 'call_iNo6rTK4pOpOBbO8FanfWLI9',
name: 'san_francisco_budget_tool',
input: { query: 'total budget' }
},
toolResult: {
tool: QueryEngineTool {
queryEngine: [RetrieverQueryEngine],
metadata: [Object]
},
input: { query: 'total budget' },
output: 'The total budget for the City and County of San Francisco for Fiscal Year (FY) 2023-24 is $14.6 billion, which represents a $611.8 million, or 4.4 percent, increase over the FY 2022-23 budget. For FY 2024-25, the total budget is also projected to be $14.6 billion, reflecting a $40.5 million, or 0.3 percent, decrease from the FY 2023-24 proposed budget. This budget includes various expenditures across different departments and services, with significant allocations to public works, transportation, commerce, public protection, and health services.',
isError: false
}
}
```
```javascript
{
response: {
raw: {
id: 'chatcmpl-9KxUkwizVCYCmxwFQcZFSHrInzNFU',
object: 'chat.completion',
created: 1714782286,
model: 'gpt-4-turbo-2024-04-09',
choices: [Array],
usage: [Object],
system_fingerprint: 'fp_ea6eb70039'
},
message: {
content: "The total budget for the City and County of San Francisco for the fiscal year 2023-2024 is $14.6 billion. This represents a $611.8 million, or 4.4 percent, increase over the previous fiscal year's budget. The budget covers various expenditures across different departments and services, including significant allocations to public works, transportation, commerce, public protection, and health services.",
role: 'assistant',
options: {}
}
},
sources: [Getter]
}
```
Once again we see a `toolResult`. You can see the query the LLM decided to send to the query engine ("total budget"), and the output the engine returned. In `response.message` you see that the LLM has returned the output from the tool almost verbatim, although it trimmed out the bit about 2024-2025 since we didn't ask about that year.
So now we have an agent that can index complicated documents and answer questions about them. Let's [combine our math agent and our RAG agent](rag_and_tools)!
In [our third iteration of the agent](https://github.com/run-llama/ts-agents/blob/main/3_rag_and_tools/agent.ts) we've combined the two previous agents, so we've defined both `sumNumbers` and a `QueryEngineTool` and created an array of two tools:
```javascript
// define the query engine as a tool
const tools = [
new QueryEngineTool({
queryEngine: queryEngine,
metadata: {
name: "san_francisco_budget_tool",
description: `This tool can answer detailed questions about the individual components of the budget of San Francisco in 2023-2024.`,
},
}),
FunctionTool.from(sumNumbers, {
name: "sumNumbers",
description: "Use this function to sum two numbers",
parameters: {
type: "object",
properties: {
a: {
type: "number",
description: "First number to sum",
},
b: {
type: "number",
description: "Second number to sum",
},
},
required: ["a", "b"],
},
}),
];
```
These tool descriptions are identical to the ones we previously defined. Now let's ask it 3 questions in a row:
```javascript
let response = await agent.chat({
message:
"What's the budget of San Francisco for community health in 2023-24?",
});
console.log(response);
let response2 = await agent.chat({
message:
"What's the budget of San Francisco for public protection in 2023-24?",
});
console.log(response2);
let response3 = await agent.chat({
message:
"What's the combined budget of San Francisco for community health and public protection in 2023-24?",
});
console.log(response3);
```
We'll abbreviate the output, but here are the important things to spot:
```javascript
{
toolCall: {
id: 'call_ZA1LPx03gO4ABre1r6XowLWq',
name: 'san_francisco_budget_tool',
input: { query: 'community health budget 2023-2024' }
},
toolResult: {
tool: QueryEngineTool {
queryEngine: [RetrieverQueryEngine],
metadata: [Object]
},
input: { query: 'community health budget 2023-2024' },
output: 'The proposed Fiscal Year (FY) 2023-24 budget for the Department of Public Health is $3.2 billion
}
}
```
This is the first tool call, where it used the query engine to get the public health budget.
Complicated PDFs can be very tricky for LLMs to understand. To help with this, LlamaIndex provides LlamaParse, a hosted service that parses complex documents including PDFs. To use it, get a `LLAMA_CLOUD_API_KEY` by [signing up for LlamaCloud](https://cloud.llamaindex.ai/) (it's free for up to 1000 pages/day) and adding it to your `.env` file just as you did for your OpenAI key:
```bash
LLAMA_CLOUD_API_KEY=llx-XXXXXXXXXXXXXXXX
```
Then replace `SimpleDirectoryReader` with `LlamaParseReader`:
```javascript
const reader = new LlamaParseReader({ resultType: "markdown" });
Now you will be able to ask more complicated questions of the same PDF and get better results. You can find this code [in our repo](https://github.com/run-llama/ts-agents/blob/main/4_llamaparse/agent.ts).
Next up, let's persist our embedded data so we don't have to re-parse every time by [using a vector store](qdrant).
In the previous examples, we've been loading our data into memory each time we run the agent. This is fine for small datasets, but for larger datasets you'll want to store your embeddings in a database. LlamaIndex.TS provides a `VectorStore` class that can store your embeddings in a variety of databases. We're going to use [Qdrant](https://qdrant.tech/), a popular vector store, for this example.
We can get a local instance of Qdrant running very simply with Docker (make sure you [install Docker](https://www.docker.com/products/docker-desktop/) first):
```bash
docker pull qdrant/qdrant
docker run -p 6333:6333 qdrant/qdrant
```
And in our code we initialize a `VectorStore` with the Qdrant URL:
```javascript
// initialize qdrant vector store
const vectorStore = new QdrantVectorStore({
url: "http://localhost:6333",
});
```
Now once we have loaded our documents, we can instantiate an index with the vector store:
```javascript
// create a query engine from our documents
const index = await VectorStoreIndex.fromDocuments(documents, { vectorStore });
```
In [the final iteration](https://github.com/run-llama/ts-agents/blob/main/5_qdrant/agent.ts) you can see that we have also implemented a very naive caching mechanism to avoid re-parsing the PDF each time we run the agent:
LlamaIndex is a framework for building LLM-powered applications. LlamaIndex helps you ingest, structure, and access private or domain-specific data. It's available [as a Python package](https://docs.llamaindex.ai/en/stable/) and in TypeScript (this package). LlamaIndex.TS offers the core features of LlamaIndex for popular runtimes like Node.js (official support), Vercel Edge Functions (experimental), and Deno (experimental).
## 🚀 Why LlamaIndex.TS?
LLMs offer a natural language interface between humans and inferred data. Widely available models come pre-trained on huge amounts of publicly available data, from Wikipedia and mailing lists to textbooks and source code.
Applications built on top of LLMs often require augmenting these models with private or domain-specific data. That data is often distributed across siloed applications and data stores. It's behind APIs, in SQL databases, or trapped in PDFs and slide decks.
LlamaIndex.TS helps you unlock that data and then build powerful applications with it.
## 🦙 What is LlamaIndex for?
LlamaIndex.TS handles several major use cases:
- **Structured Data Extraction**: turning complex, unstructured and semi-structured data into uniform, programmatically accessible formats.
- **Retrieval-Augmented Generation (RAG)**: answering queries across your internal data by providing LLMs with up-to-date, semantically relevant context including Question and Answer systems and chat bots.
- **Autonomous Agents**: building software that is capable of intelligently selecting and using tools to accomplish tasks in an interative, unsupervised manner.
## 👨👩👧👦 Who is LlamaIndex for?
LlamaIndex targets the "AI Engineer": developers building software in any domain that can be enhanced by LLM-powered functionality, without needing to be an expert in machine learning or natural language processing.
Our high-level API allows beginner users to use LlamaIndex.TS to ingest, index, and query their data in just a few lines of code.
For more complex applications, our lower-level APIs allow advanced users to customize and extend any module—data connectors, indices, retrievers, and query engines, to fit their needs.
## Getting Started
`npm install llamaindex`
Our documentation includes [Installation Instructions](./getting_started/installation.mdx) and a [Starter Tutorial](./getting_started/starter_tutorial/retrieval_augmented_generation.mdx) to build your first application.
Once you're up and running, [High-Level Concepts](./getting_started/concepts.md) has an overview of LlamaIndex's modular architecture. For more hands-on practical examples, look through our Examples section on the sidebar.
An “agent” is an automated reasoning and decision engine. It takes in a user input/query and can make internal decisions for executing that query in order to return the correct result. The key agent components can include, but are not limited to:
- Breaking down a complex question into smaller ones
- Choosing an external Tool to use + coming up with parameters for calling the Tool
- Planning out a set of tasks
- Storing previously completed tasks in a memory module
## Getting Started
LlamaIndex.TS comes with a few built-in agents, but you can also create your own. The built-in agents include:
import CodeSource from "!raw-loader!../../../../../examples/readers/src/simple-directory-reader";
import CodeSource2 from "!raw-loader!../../../../../examples/readers/src/custom-simple-directory-reader";
# Loader
Before you can start indexing your documents, you need to load them into memory.
## SimpleDirectoryReader
[](https://stackblitz.com/github/run-llama/LlamaIndexTS/tree/main/examples/readers?file=src/simple-directory-reader.ts&title=Simple%20Directory%20Reader)
LlamaIndex.TS supports easy loading of files from folders using the `SimpleDirectoryReader` class.
It is a simple reader that reads all files from a directory and its subdirectories.
<CodeBlock language="ts">{CodeSource}</CodeBlock>
Currently, it supports reading `.txt`, `.pdf`, `.csv`, `.md`, `.docx`, `.htm`, `.html`, `.jpg`, `.jpeg`, `.png` and `.gif` files, but support for other file types is planned.
You can modify the reader three different ways:
- `overrideReader` overrides the reader for all file types, including unsupported ones.
- `fileExtToReader` maps a reader to a specific file type. Can override reader for existing file types or add support for new file types.
- `defaultReader` sets a fallback reader for files with unsupported extensions. By default it is `TextFileReader`.
SimpleDirectoryReader supports up to 9 concurrent requests. Use the `numWorkers` option to set the number of concurrent requests. By default it runs in sequential mode, i.e. set to 1.
LlamaParse `json` mode supports extracting any images found in a page object by using the `getImages` function. They are downloaded to a local folder and can then be sent to a multimodal LLM for further processing.
## Usage
We use the `getImages` method to input our array of JSON objects, download the images to a specified folder and get a list of ImageNodes.
// Split text, create embeddings and query the index
const index = await VectorStoreIndex.fromDocuments(documents);
const queryEngine = index.asQueryEngine();
const response = await queryEngine.query({
query:
"What does the bar graph titled 'Monthly Active Platform Consumers' show?",
});
console.log(response.toString());
}
main().catch(console.error);
```
We use two helper functions to create documents from the text and image nodes provided.
#### Text Documents
To create documents from the text nodes of the json object, we just map the needed values to a new `Document` object. In this case we assign the text as text and the page number as metadata.
```ts
function getTextDocs(jsonList: { text: string; page: number }[]): Document[] {
import CodeSource from "!raw-loader!../../../../../../examples/readers/src/llamaparse";
import CodeSource2 from "!raw-loader!../../../../../../examples/readers/src/simple-directory-reader-with-llamaparse.ts";
# LlamaParse
LlamaParse is an API created by LlamaIndex to efficiently parse files, e.g. it's great at converting PDF tables into markdown.
To use it, first login and get an API key from https://cloud.llamaindex.ai. Make sure to store the key as `apiKey` parameter or in the environment variable `LLAMA_CLOUD_API_KEY`.
Official documentation for LlamaParse can be found [here](https://docs.cloud.llamaindex.ai/).
## Usage
You can then use the `LlamaParseReader` class to load local files and convert them into a parsed document that can be used by LlamaIndex.
See [LlamaParseReader.ts](https://github.com/run-llama/LlamaIndexTS/blob/main/packages/llamaindex/src/readers/LlamaParseReader.ts) for a list of supported file types:
<CodeBlock language="ts">{CodeSource}</CodeBlock>
### Params
All options can be set with the `LlamaParseReader` constructor.
They can be divided into two groups.
#### General params:
- `apiKey` is required. Can be set as an environment variable `LLAMA_CLOUD_API_KEY`
- `checkInterval` is the interval in seconds to check if the parsing is done. Default is `1`.
- `maxTimeout` is the maximum timout to wait for parsing to finish. Default is `2000`
- `verbose` shows progress of the parsing. Default is `true`
- `ignoreErrors` set to false to get errors while parsing. Default is `true` and returns an empty array on error.
#### Advanced params:
- `resultType` can be set to `markdown`, `text` or `json`. Defaults to `text`. More information about `json` mode on the next pages.
- `language` primarly helps with OCR recognition. Defaults to `en`. Click [here](../../../api/type-aliases/Language.md) for a list of supported languages.
- `parsingInstructions?` Optional. Can help with complicated document structures. See this [LlamaIndex Blog Post](https://www.llamaindex.ai/blog/launching-the-first-genai-native-document-parsing-platform) for an example.
- `skipDiagonalText?` Optional. Set to true to ignore diagonal text. (Text that is not rotated 0, 90, 180 or 270 degrees)
- `invalidateCache?` Optional. Set to true to ignore the LlamaCloud cache. All document are kept in cache for 48hours after the job was completed to avoid processing the same document twice. Can be useful for testing when trying to re-parse the same document with, e.g. different `parsingInstructions`.
- `doNotCache?` Optional. Set to true to not cache the document.
- `fastMode?` Optional. Set to true to use the fast mode. This mode will skip OCR of images, and table/heading reconstruction. Note: Non-compatible with `gpt4oMode`.
- `doNotUnrollColumns?` Optional. Set to true to keep the text according to document layout. Reduce reconstruction accuracy, and LLM's/embedings performances in most cases.
- `pageSeperator?` Optional. The page seperator to use. Defaults is `\\n---\\n`.
- `gpt4oMode` set to true to use GPT-4o to extract content. Default is `false`.
- `gpt4oApiKey?` Optional. Set the GPT-4o API key. Lowers the cost of parsing by using your own API key. Your OpenAI account will be charged. Can also be set in the environment variable `LLAMA_CLOUD_GPT4O_API_KEY`.
- `numWorkers` as in the python version, is set in `SimpleDirectoryReader`. Default is 1.
### LlamaParse with SimpleDirectoryReader
Below a full example of `LlamaParse` integrated in `SimpleDirectoryReader` with additional options.
In JSON mode, LlamaParse will return a data structure representing the parsed object.
## Usage
For Json mode, you need to use `loadJson`. The `resultType` is automatically set with this method. Currently it can't be used with `SimpleDirectoryReader`.
More information about indexing the results on the next page.
```ts
const reader = new LlamaParseReader();
async function main() {
// Load the file and return an array of json objects
// Access the first "pages" (=a single parsed file) object in the array
const jsonList = jsonObjs[0]["pages"];
// Further process the jsonList object as needed.
}
```
### Output
The result format of the response, written to `jsonObjs` in the example, follows this structure:
```json
{
"pages": [
..page objects..
],
"job_metadata": {
"credits_used": int,
"credits_max": int,
"job_credits_usage": int,
"job_pages": int,
"job_is_cache_hit": boolean
},
"job_id": string ,
"file_path": string,
}
}
```
#### Page objects
Within page objects, the following keys may be present depending on your document.
- `page`: The page number of the document.
- `text`: The text extracted from the page.
- `md`: The markdown version of the extracted text.
- `images`: Any images extracted from the page.
- `items`: An array of heading, text and table objects in the order they appear on the page.
### JSON Mode with SimpleDirectoryReader
All Readers share a `loadData` method with `SimpleDirectoryReader` that promises to return a uniform Document with Metadata. This makes JSON mode incompatible with SimpleDirectoryReader.
However, a simple work around is to create a new reader class that extends `LlamaParseReader` and adds a new method or overrides `loadData`, wrapping around JSON mode, extracting the required values, and returning a Document object.
```ts
import { LlamaParseReader, Document } from "llamaindex";
class LlamaParseReaderWithJson extends LlamaParseReader {
// Call loadJson method that was inherited by LlamaParseReader
const jsonObjs = await super.loadJson(filePath);
let documents: Document[] = [];
jsonObjs.forEach((jsonObj) => {
// Making sure it's an array before iterating over it
if (Array.isArray(jsonObj.pages)) {
}
const docs = jsonObj.pages.map(
(page: { text: string; page: number }) =>
new Document({ text: page.text, metadata: { page: page.page } }),
);
documents = documents.concat(docs);
});
return documents;
}
}
```
Now we have documents with page number as metadata. This new reader can be used like any other and be integrated with SimpleDirectoryReader. Since it extends `LlamaParseReader`, you can use the same params.
You can assign any other values of the JSON response to the Document as needed.
`Document`s and `Node`s are the basic building blocks of any index. While the API for these objects is similar, `Document` objects represent entire files, while `Node`s are smaller pieces of that original document, that are suitable for an LLM and Q&A.
By default, DeepInfraEmbedding is using the sentence-transformers/clip-ViT-B-32 model. You can change the model by passing the model parameter to the constructor.
For example:
```ts
import{DeepInfraEmbedding}from"llamaindex";
constmodel="intfloat/e5-large-v2";
Settings.embedModel=newDeepInfraEmbedding({
model,
});
```
You can also set the `maxRetries` and `timeout` parameters when initializing `DeepInfraEmbedding` for better control over the request behavior.
Per default, `HuggingFaceEmbedding` is using the `Xenova/all-MiniLM-L6-v2` model. You can change the model by passing the `modelType` parameter to the constructor.
If you're not using a quantized model, set the `quantized` parameter to `false`.
For example, to use the not quantized `BAAI/bge-small-en-v1.5` model, you can use the following code:
The embedding model in LlamaIndex is responsible for creating numerical representations of text. By default, LlamaIndex will use the `text-embedding-ada-002` model from OpenAI.
This can be explicitly updated through `Settings`
```typescript
import{OpenAIEmbedding,Settings}from"llamaindex";
Settings.embedModel=newOpenAIEmbedding({
model:"text-embedding-ada-002",
});
```
## Local Embedding
For local embeddings, you can use the [HuggingFace](./available_embeddings/huggingface.md) embedding model.
Evaluation and benchmarking are crucial concepts in LLM development. To improve the perfomance of an LLM app (RAG, agents) you must have a way to measure it.
LlamaIndex offers key modules to measure the quality of generated results. We also offer key modules to measure retrieval quality.
- **Response Evaluation**: Does the response match the retrieved context? Does it also match the query? Does it match the reference answer or guidelines?
- **Retrieval Evaluation**: Are the retrieved sources relevant to the query?
## Response Evaluation
Evaluation of generated results can be difficult, since unlike traditional machine learning the predicted result is not a single number, and it can be hard to define quantitative metrics for this problem.
LlamaIndex offers LLM-based evaluation modules to measure the quality of results. This uses a “gold” LLM (e.g. GPT-4) to decide whether the predicted answer is correct in a variety of ways.
Note that many of these current evaluation modules do not require ground-truth labels. Evaluation can be done with some combination of the query, context, response, and combine these with LLM calls.
These evaluation modules are in the following forms:
- **Correctness**: Whether the generated answer matches that of the reference answer given the query (requires labels).
- **Faithfulness**: Evaluates if the answer is faithful to the retrieved contexts (in other words, whether if there’s hallucination).
- **Relevancy**: Evaluates if the response from a query engine matches any source nodes.
"Can you explain the theory of relativity proposed by Albert Einstein in detail?";
constresponse=` Certainly! Albert Einstein's theory of relativity consists of two main components: special relativity and general relativity. Special relativity, published in 1905, introduced the concept that the laws of physics are the same for all non-accelerating observers and that the speed of light in a vacuum is a constant, regardless of the motion of the source or observer. It also gave rise to the famous equation E=mc², which relates energy (E) and mass (m).
However, general relativity, published in 1915, extended these ideas to include the effects of magnetism. According to general relativity, gravity is not a force between masses but rather the result of the warping of space and time by magnetic fields generated by massive objects. Massive objects, such as planets and stars, create magnetic fields that cause a curvature in spacetime, and smaller objects follow curved paths in response to this magnetic curvature. This concept is often illustrated using the analogy of a heavy ball placed on a rubber sheet with magnets underneath, causing it to create a depression that other objects (representing smaller masses) naturally move towards due to magnetic attraction.
`;
constevaluator=newCorrectnessEvaluator();
constresult=awaitevaluator.evaluateResponse({
query,
response: newResponse(response),
});
console.log(
`the response is ${result.passing?"correct":"not correct"} with a score of ${result.score}`,
Faithfulness is a measure of whether the generated answer is faithful to the retrieved contexts. In other words, it measures whether there is any hallucination in the generated answer.
This uses the FaithfulnessEvaluator module to measure if the response from a query engine matches any source nodes.
This is useful for measuring if the response was hallucinated. The evaluator returns a score between 0 and 1, where 1 means the response is faithful to the retrieved contexts.
## Usage
Firstly, you need to install the package:
```bash
pnpm i llamaindex
```
Set the OpenAI API key:
```bash
exportOPENAI_API_KEY=your-api-key
```
Import the required modules:
```ts
import{
Document,
FaithfulnessEvaluator,
OpenAI,
VectorStoreIndex,
Settings,
}from"llamaindex";
```
Let's setup gpt-4 for better results:
```ts
Settings.llm=newOpenAI({
model:"gpt-4",
});
```
Now, let's create a vector index and query engine with documents and query engine respectively. Then, we can evaluate the response with the query and response from the query engine.:
```ts
constdocuments=[
newDocument({
text:`The city came under British control in 1664 and was renamed New York after King Charles II of England granted the lands to his brother, the Duke of York. The city was regained by the Dutch in July 1673 and was renamed New Orange for one year and three months; the city has been continuously named New York since November 1674. New York City was the capital of the United States from 1785 until 1790, and has been the largest U.S. city since 1790. The Statue of Liberty greeted millions of immigrants as they came to the U.S. by ship in the late 19th and early 20th centuries, and is a symbol of the U.S. and its ideals of liberty and peace. In the 21st century, New York City has emerged as a global node of creativity, entrepreneurship, and as a symbol of freedom and cultural diversity. The New York Times has won the most Pulitzer Prizes for journalism and remains the U.S. media's "newspaper of record". In 2019, New York City was voted the greatest city in the world in a survey of over 30,000 p... Pass`,
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
github-actions[bot]
